Biological image transformation using machine-learning models

ABSTRACT

Described are systems and methods for training a machine-learning model to generate image of biological samples, and systems and methods for generating enhanced images of biological samples. The method for training a machine-learning model to generate images of biological samples may include obtaining a plurality of training images comprising a training image of a first type, and a training image of a second type. The method may also include generating, based on the training image of the first type, a plurality of wavelet coefficients using the machine-learning model; generating, based on the plurality of wavelet coefficients, a synthetic image of the second type; comparing the synthetic image of the second type with the training image of the second type; and updating the machine-learning model based on the comparison.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application No.63/075,751, filed Sep. 8, 2020, and U.S. Provisional Application No.63/143,707, filed Jan. 29, 2021, the contents of which are incorporatedby reference herein in their entirety.

FIELD OF THE INVENTION

The present disclosure relates generally to machine-learning techniques,and more specifically to low-cost, machine-learning-based generation ofimage data at scale. The generated image data (e.g., image data ofbiological samples) can provide sufficient richness and depth optimizedfor downstream processing (e.g., phenotyping). Some embodiments of thesystem comprise a programmable spatial light modulator (“SLM”) toproduce data optimized for downstream processing (e.g., phenotyping) ata high speed without mechanical modifications to the system. Someembodiments of the system comprise a machine-learning model with anattention layer comprising a plurality of weights corresponding to aplurality of illumination settings (e.g., different illuminationemitters of an illumination source) for identifying an optimalillumination pattern for capturing the image data. Some embodiments ofthe system comprise techniques for evaluating candidate treatments withrespect to a disease of interest.

BACKGROUND

Bright-field images of biological samples can be obtained at scale andat low cost due to inexpensive equipment, ease of clinical deployment,and low processing and storage resource requirements for capturedimages. Obtaining bright-field images is generally non-invasive andinvolves low photo-toxicity. However, low-contrast images lack richvisual details, thus making them unsuitable for many downstream analyses(e.g., phenotypic exploration of microscopic samples). In comparison,other image modalities (e.g., fluorescence images) can provide richvisual information of the captured samples. However, obtainingfluorescence images requires additional equipment and materials and canbe time-consuming and computing resource intensive. Thus, fluorescenceimages can be difficult to obtain at scale and in a low-cost manner.

Transforming bright-field images of biological samples into enhanced,high-quality images can be difficult for a number of reasons. First, thebright-field images suffer from the inherent class imbalance problem(i.e., abundant low frequency signals but fewer high frequency signals).Further, the overall geometry of the bright-field images needs to beextracted and maintained through the transformation. Furthermore, manyfactors such as the illumination pattern under which the bright-fieldimages are taken can impact the effectiveness of the transformation.Further still, the robustness of the transformation in supportingdownstream analyses needs to be quantified and validated.

BRIEF SUMMARY

Described are systems and methods for training machine-learning modelsto generate images of biological samples. Also described are systems andmethods for generating enhanced images of biological samples. Thesystems and methods can be used, for example to obtain images of a firsttype, for example, a bright-field image type. The obtained images of thefirst image type can then be used by the systems and methods to generatea synthetic image of a second type, for example, a fluorescence image.

In some embodiments, a method for training a machine-learning model togenerate images of biological samples, comprises obtaining a pluralityof training images. The plurality of images comprises a training imageof a first type, and a training image of a second type. The method alsocomprises generating, based on the training image of the first type, aplurality of wavelet coefficients using the machine-learning model;generating, based on the plurality of wavelet coefficients, a syntheticimage of the second type; comparing the synthetic image of the secondtype with the training image of the second type; and updating themachine-learning model based on the comparison.

In some embodiment, a method for generating enhanced images ofbiological samples, comprises obtaining, using a microscope, an image ofa biological sample; and generating, based on the image, an enhancedimage of the biological sample using a machine-learning model. Themachine-learning model may be trained by: obtaining a plurality oftraining images comprising a training image of a first type, and atraining image of a second type; generating, based on the training imageof the first type, a plurality of wavelet coefficients using themachine-learning model; generating, based on the plurality of waveletcoefficients, a synthetic image of the second type; comparing thesynthetic image of the second type with the training image of the secondtype; and updating the machine-learning model based on the comparison.

In some embodiments, a system for training a machine-learning model togenerate images of biological samples, comprises: a computing systemcomprising one or more processors, and one or more memories storing amachine-learning model, wherein the computing system is configured toreceive a plurality of training images of a first type and one atraining image of a second type. The computing system may be configuredto: generate, based on the training images of the first type, aplurality of wavelet coefficients using the machine-learning model;generate, based on the plurality of wavelet coefficients, a syntheticimage of the second type; compare the synthetic image of the second typewith the training image of the second type; and update themachine-learning model based on the comparison.

In some embodiments, a system for generating enhanced images ofbiological samples, comprises: a computing system comprising one or moreprocessors, and one or more memories storing a machine-learning model.The computing system may be configured to receive an image of abiological sample obtained from a microscope and generate, based on theimage, an enhanced image of the biological sample using amachine-learning model. The machine-learning model may be been trainedby: obtaining a plurality of training images comprising a training imageof a first type, and a training image of a second type; generating,based on the training image of the first type, a plurality of waveletcoefficients using the machine-learning model; generating, based on theplurality of wavelet coefficients, a synthetic image of the second type;comparing the synthetic image of the second type with the training imageof the second type; and updating the machine-learning model based on thecomparison.

An exemplary method for training a machine-learning model to generateimages of biological samples comprises: obtaining a plurality oftraining images comprising: a training image of a first type, and atraining image of a second type; generating, based on the training imageof the first type, a plurality of wavelet coefficients using themachine-learning model; generating, based on the plurality of waveletcoefficients, a synthetic image of the second type; comparing thesynthetic image of the second type with the training image of the secondtype; and updating the machine-learning model based on the comparison.

In some embodiments, the training image of the first type is abright-field image of a biological sample.

In some embodiments, the training image of the second type is afluorescence image of the biological sample.

In some embodiments, the machine-learning model comprises a generatorand a discriminator.

In some embodiments, the machine-learning model comprises a conditionalGAN model.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the plurality of neural networks comprises aplurality of U-Net neural networks.

In some embodiments, the discriminator is a PatchGAN neural network.

In some embodiments, the method further comprises: generating, based onthe training image of the first type, an image of a third type.

In some embodiments, the image of the third type is a phase shift image.

In some embodiments, the method further comprises: generating, based onthe training image of the first type, an image of a fourth type.

In some embodiments, the image of the fourth type comprises segmentationdata.

In some embodiments, the training image of the first type is capturedusing a microscope according to a first illumination scheme.

In some embodiments, the first illumination scheme comprises one or moreillumination patterns.

In some embodiments, the training image of the first type is part of abright-field image array.

In some embodiments, the plurality of training images is a firstplurality of training images, the method further comprising: based onthe comparison, identifying a second illumination scheme; obtaining asecond plurality of training images comprising one or more images of thefirst type, wherein the one or more images of the first type areobtained based on the second illumination scheme; training themachine-learning model based on the second plurality of training images.

In some embodiments, the method further comprises obtaining, using amicroscope, a plurality of images of the first type; and generating,based on the obtained plurality of images, a plurality of syntheticimages of the second type using the machine-learning model.

In some embodiments, the method further comprises training a classifierbased on the plurality of synthetic images of the second type.

In some embodiments, the microscope is a first microscope, wherein theclassifier is a first classifier, further comprising: obtaining, using asecond microscope, a plurality of images of the second type; training asecond classifier based on the plurality of images of the second type;comparing performance of the first classifier and the second classifier.

In some embodiments, the second microscope is a fluorescence microscope.

An exemplary method for generating enhanced images of biological samplescomprises: obtaining, using a microscope, an image of a biologicalsample; and generating, based on the image, an enhanced image of thebiological sample using a machine-learning model, wherein themachine-learning model has been trained by: obtaining a plurality oftraining images comprising a training image of a first type, and atraining image of a second type; generating, based on the training imageof the first type, a plurality of wavelet coefficients using themachine-learning model; generating, based on the plurality of waveletcoefficients, a synthetic image of the second type; comparing thesynthetic image of the second type with the training image of the secondtype; and updating the machine-learning model based on the comparison.

In some embodiments, the training image of the first type is abright-field image of a biological sample.

In some embodiments, the training image of the second type is afluorescence image of the biological sample.

In some embodiments, the machine-learning model comprises a generatorand a discriminator.

In some embodiments, the machine-learning model comprises a conditionalGAN model.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the plurality of neural networks comprises aplurality of U-Net neural networks.

In some embodiments, the discriminator is a PatchGAN neural network.

In some embodiments, the method further comprises: generating, based onthe training image of the first type, an image of a third type.

In some embodiments, the image of the third type is a phase shift image.

In some embodiments, the method further comprises: generating, based onthe training image of the first type, an image of a fourth type.

In some embodiments, the image of the fourth type comprises segmentationdata.

In some embodiments, the training image of the first type is capturedusing a microscope according to a first illumination scheme.

In some embodiments, the first illumination scheme comprises one or moreillumination patterns.

In some embodiments, the training image of the first type is part of abright-field image array.

In some embodiments, the plurality of training images is a firstplurality of training images, the method further comprising: based onthe comparison, identifying a second illumination scheme; obtaining asecond plurality of training images comprising one or more images of thefirst type, wherein the one or more images of the first type areobtained based on the second illumination scheme; training themachine-learning model based on the second plurality of training images.

In some embodiments, the method further comprises obtaining, using amicroscope, a plurality of images of the first type; and generating,based on the obtained plurality of images, a plurality of syntheticimages of the second type using the machine-learning model.

In some embodiments, the method further comprises: training a classifierbased on the plurality of synthetic images of the second type.

In some embodiments, the microscope is a first microscope, wherein theclassifier is a first classifier, further comprising: obtaining, using asecond microscope, a plurality of images of the second type; training asecond classifier based on the plurality of images of the second type;comparing performance of the first classifier and the second classifier.

In some embodiments, the second microscope is a fluorescence microscope.

An exemplary system for training a machine-learning model to generateimages of biological samples comprises: a computing system comprisingone or more processors, and one or more memories storing amachine-learning model, wherein the computing system is configured toreceive a plurality of training images of a first type and one atraining image of a second type, and wherein the computing system isconfigured to generate, based on the training images of the first type,a plurality of wavelet coefficients using the machine-learning model;generate, based on the plurality of wavelet coefficients, a syntheticimage of the second type; compare the synthetic image of the second typewith the training image of the second type; and update themachine-learning model based on the comparison.

In some embodiments, the training image of the first type is abright-field image of a biological sample.

In some embodiments, the training image of the second type is afluorescence image of the biological sample.

In some embodiments, the machine-learning model comprises a generatorand a discriminator.

In some embodiments, the machine-learning model comprises a conditionalGAN model.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the plurality of neural networks comprises aplurality of U-Net neural networks.

In some embodiments, the discriminator is a PatchGAN neural network.

In some embodiments, the computing system is further configured to:generate, based on the training image of the first type, an image of athird type.

In some embodiments, the image of the third type is a phase shift image.

In some embodiments, the computing system is further configured to:generate, based on the training image of the first type, an image of afourth type.

In some embodiments, the image of the fourth type comprises segmentationdata.

In some embodiments, the training image of the first type is capturedusing a microscope according to a first illumination scheme.

In some embodiments, the first illumination scheme comprises one or moreillumination patterns.

In some embodiments, the training image of the first type is part of abright-field image array.

In some embodiments, the plurality of training images is a firstplurality of training images, and wherein the computing system isfurther configured to: based on the comparison, identify a secondillumination scheme; obtain a second plurality of training imagescomprising one or more images of the first type, wherein the one or moreimages of the first type are obtained based on the second illuminationscheme; train the machine-learning model based on the second pluralityof training images.

In some embodiments, the computing system is further configured to:obtain, using a microscope, a plurality of images of the first type; andgenerate, based on the obtained plurality of images, a plurality ofsynthetic images of the second type using the machine-learning model.

In some embodiments, the computing system is further configured to:train a classifier based on the plurality of synthetic images of thesecond type.

In some embodiments, the microscope is a first microscope, wherein theclassifier is a first classifier, wherein the computing system isfurther configured to: obtain, using a second microscope, a plurality ofimages of the second type; train a second classifier based on theplurality of images of the second type; compare performance of the firstclassifier and the second classifier.

In some embodiments, the second microscope is a fluorescence microscope.

An exemplary system for generating enhanced images of biological samplescomprises: a computing system comprising one or more processors, and oneor more memories storing a machine-learning model, wherein the computingsystem is configured to receive an image of a biological sample obtainedfrom a microscope and generate, based on the image, an enhanced image ofthe biological sample using a machine-learning model, wherein themachine-learning model has been trained by: obtaining a plurality oftraining images comprising a training image of a first type, and atraining image of a second type; generating, based on the training imageof the first type, a plurality of wavelet coefficients using themachine-learning model; generating, based on the plurality of waveletcoefficients, a synthetic image of the second type; comparing thesynthetic image of the second type with the training image of the secondtype; and updating the machine-learning model based on the comparison.

In some embodiments, the training image of the first type is abright-field image of a biological sample.

In some embodiments, the training image of the second type is afluorescence image of the biological sample.

In some embodiments, the machine-learning model comprises a generatorand a discriminator.

In some embodiments, the machine-learning model comprises a conditionalGAN model.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the plurality of neural networks comprises aplurality of U-Net neural networks.

In some embodiments, the discriminator is a PatchGAN neural network.

In some embodiments, the machine learning model is further trained bygenerating, based on the training image of the first type, an image of athird type.

In some embodiments, the image of the third type is a phase shift image.

In some embodiments, the machine-learning model has been trained by:generating, based on the training image of the first type, an image of afourth type.

In some embodiments, the image of the fourth type comprises segmentationdata.

In some embodiments, the training image of the first type is capturedusing a microscope according to a first illumination scheme.

In some embodiments, the first illumination scheme comprises one or moreillumination patterns.

In some embodiments, the training image of the first type is part of abright-field image array.

In some embodiments, the plurality of training images is a firstplurality of training images, wherein the machine-learning model hasbeen trained by: based on the comparison, identifying a secondillumination scheme; obtaining a second plurality of training imagescomprising one or more images of the first type, wherein the one or moreimages of the first type are obtained based on the second illuminationscheme; training the machine-learning model based on the secondplurality of training images.

In some embodiments, the machine-learning model has been trained by:obtaining, using a microscope, a plurality of images of the first type;and generating, based on the obtained plurality of images, a pluralityof synthetic images of the second type using the machine-learning model.

In some embodiments, the machine-learning model has been trained by:training a classifier based on the plurality of synthetic images of thesecond type.

In some embodiments, the microscope is a first microscope, wherein theclassifier is a first classifier, wherein the machine-learning model hasbeen trained by: obtaining, using a second microscope, a plurality ofimages of the second type; training a second classifier based on theplurality of images of the second type; comparing performance of thefirst classifier and the second classifier.

In some embodiments, the second microscope is a fluorescence microscope.

An exemplary method of processing images of a biological sample toobtain one or more output images comprises: obtaining a plurality ofimages of the biological sample using a plurality of configurations of aSLM of an optical system, wherein the SLM is located in an optical pathbetween the biological sample and an image recording device; andinputting the plurality of images of the biological sample into atrained machine-learning model to obtain the one or more outputs images.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to generate one or more opticalaberrations.

In some embodiments, generating one or more optical aberrationscomprises a spherical aberration, astigmatism, defocus, distortion,tilt, or any combination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to enhance one or more features.

In some embodiments, the one or more features comprise a cell border, anactin filament, nuclear shape, cytoplasm segmentation, or anycombination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to reduce optical aberrations.

In some embodiments, the plurality of SLM configurations is to obtainimages of the biological sample at different depths.

In some embodiments, the machine-learning model is configured togenerate, based on an image of a first type, an image of a second type.

In some embodiments, the first type of images are bright-field images.

In some embodiments, the second type of images are fluorescence images.

In some embodiments, the second type of images are enhanced versions ofthe first type of images.

In some embodiments, the machine-learning model is a GAN model or aself-supervised model.

In some embodiments, the plurality of images are obtained using aplurality of configurations of a light source of the optical system.

In some embodiments, the light source is a LED array of the opticalsystem.

In some embodiments, at least one configuration of the plurality of SLMconfigurations is obtained by: training the machine-learning model;evaluating the trained machine-learning model; and identifying the atleast one configuration based on the evaluation.

In some embodiments, the trained machine-learning model is configured toreceive an input image and output an enhanced version of the inputimage.

In some embodiments, the enhanced version of the input image comprisesone or more enhanced cellular phenotypes.

An exemplary electronic device for processing images of a biologicalsample to obtain one or more output images comprises: one or moreprocessors; a memory; and one or more programs, wherein the one or moreprograms are stored in the memory and configured to be executed by theone or more processors, the one or more programs including instructionsfor: obtaining a plurality of images of the biological sample using aplurality of configurations of a SLM of an optical system, wherein theSLM is located in an optical path between the biological sample and animage recording device; and inputting the plurality of images of thebiological sample into a trained machine-learning model to obtain theone or more output images.

An exemplary non-transitory computer-readable storage medium stores oneor more programs for processing images of a biological sample to obtainone or more output images, the one or more programs comprisinginstructions, which when executed by one or more processors of anelectronic device, cause the electronic device to: obtain a plurality ofimages of the biological sample using a plurality of configurations of aSLM of an optical system, wherein the SLM is located in an optical pathbetween the biological sample and an image recording device; and inputthe plurality of images of the biological sample into a trainedmachine-learning model to obtain the one or more output images.

An exemplary method of classifying images of a biological samplecomprises: obtaining a plurality of images of the biological sampleusing a plurality of configurations of an SLM of an optical system,wherein the SLM is located in an optical path between the biologicalsample and an image recording device; and inputting the plurality ofimages of the biological sample into a trained machine-learning model toobtain one or more classification outputs.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to generate one or more opticalaberrations.

In some embodiments, generating one or more optical aberrationscomprises a spherical aberration, astigmatism, defocus, distortion,tilt, or any combination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to enhance one or more features.

In some embodiments, the one or more features comprise a cell border, anactin filament, nuclear shape, cytoplasm segmentation, or anycombination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to reduce optical aberrations.

In some embodiments, the plurality of SLM configurations is to obtainimages of the biological sample at different depths.

In some embodiments, the plurality of images are obtained using aplurality of configurations of a light source of the optical system.

In some embodiments, the light source is a LED array of the opticalsystem.

In some embodiments, at least one configuration of the plurality of SLMconfigurations is obtained by: training the machine-learning model;evaluating the trained machine-learning model; and identifying the atleast one configuration based on the evaluation.

In some embodiments, the trained machine-learning model is configured toreceive an input image and detect one or more pre-defined objects in theinput image.

In some embodiments, the pre-defined objects include a diseased tissue.

An exemplary electronic device for classifying images of a biologicalsample comprises: one or more processors; a memory; and one or moreprograms, wherein the one or more programs are stored in the memory andconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: obtaining a plurality of images ofthe biological sample using a plurality of configurations of an SLM ofan optical system, wherein the SLM is located in an optical path betweenthe biological sample and an image recording device; and inputting theplurality of images of the biological sample into a trainedmachine-learning model to obtain one or more classification outputs.

An exemplary non-transitory computer-readable storage medium stores oneor more programs for classifying images of a biological sample, the oneor more programs comprising instructions, which when executed by one ormore processors of an electronic device, cause the electronic device to:obtain a plurality of images of the biological sample using a pluralityof configurations of an SLM of an optical system, wherein the SLM islocated in an optical path between the biological sample and an imagerecording device; and input the plurality of images of the biologicalsample into a trained machine-learning model to obtain one or moreclassification outputs.

An exemplary method for training a machine-learning model comprises:obtaining a plurality of images of a biological sample using a pluralityof configurations of an SLM of an optical system, wherein the SLM islocated in an optical path between the biological sample and an imagerecording device; and training the machine-learning model using theplurality of images.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to generate one or more opticalaberrations.

In some embodiments, generating one or more optical aberrationscomprises a spherical aberration, astigmatism, defocus, distortion,tilt, or any combination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to enhance one or more features.

In some embodiments, the one or more features comprise a cell border, anactin filament, nuclear shape, cytoplasm segmentation, or anycombination thereof.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to reduce optical aberrations.

In some embodiments, at least one configuration of the plurality ofconfigurations of the SLM is to obtain images of the biological sampleat different depths.

In some embodiments, the machine-learning model is configured togenerate, based on an image of a first type, an image of a second type.

In some embodiments, the first type of images are bright-field images.

In some embodiments, the second type of images are fluorescence images.

In some embodiments, the machine-learning model is a GAN model or aself-supervised model.

In some embodiments, the machine-learning model is a classificationmodel.

In some embodiments, the plurality of images are obtained using aplurality of configurations of a light source of the optical system.

In some embodiments, the light source is a LED array of the opticalsystem.

In some embodiments, training the machine-learning model comprises: (a)training the machine-learning model using a first image, wherein thefirst image is obtained using a first configuration of the SLM of theoptical system; (b) evaluating the trained machine-learning model; (c)based on the evaluation, identifying a second configuration of the SLM;and (d) training the machine-learning model using a second image,wherein the second image is obtained using the second configuration ofthe SLM of the optical system.

In some embodiments, the evaluation is based on a loss function of themachine-learning model.

In some embodiments, the method further comprises: repeating steps(a)-(d) until a threshold is met.

In some embodiments, the threshold is indicative of convergence of thetraining.

In some embodiments, the trained machine-learning model is configured toreceive an input image and output an enhanced version of the inputimage.

In some embodiments, the enhanced version of the input image comprisesone or more enhanced cellular phenotypes.

In some embodiments, the trained machine-learning model is configured toreceive an input image and detect one or more pre-defined objects in theinput image.

In some embodiments, the pre-defined objects include a diseased tissue.

An exemplary electronic device for training a machine-learning modelcomprises: one or more processors; a memory; and one or more programs,wherein the one or more programs are stored in the memory and configuredto be executed by the one or more processors, the one or more programsincluding instructions for: obtaining a plurality of images of abiological sample using a plurality of configurations of an SLM of anoptical system, wherein the SLM is located in an optical path betweenthe biological sample and an image recording device; and training themachine-learning model using the plurality of images.

An exemplary non-transitory computer-readable storage medium stores oneor more programs for training a machine-learning model, the one or moreprograms comprising instructions, which when executed by one or moreprocessors of an electronic device, cause the electronic device to:obtain a plurality of images of a biological sample using a plurality ofconfigurations of an SLM of an optical system, wherein the SLM islocated in an optical path between the biological sample and an imagerecording device; and train the machine-learning model using theplurality of images.

An exemplary method of generating enhanced images of biological samplescomprises: obtaining, using a microscope, an image of a biologicalsample illuminated using an illumination pattern of an illuminationsource, wherein the illumination pattern is determined by: training aclassification model configured to receive an input image and output aclassification result, training, using the trained classification model,a machine-learning model having an plurality of weights corresponding toa plurality of illumination settings, and identifying the illuminationpattern based on the plurality of weights of the trainedmachine-learning model; and generating an enhanced image of thebiological sample by inputting the obtained image of the biologicalsample into the trained machine-learning model.

In some embodiments, the obtained image is a bright-field image.

In some embodiments, the enhanced image is a fluorescence image, a phaseimage, or a combination thereof.

In some embodiments, the illumination source comprises an array ofillumination emitters.

In some embodiments, the illumination source is a LED array.

In some embodiments, the illumination pattern indicates whether eachillumination emitter is turned on or off and the intensity of eachillumination emitter.

In some embodiments, each illumination setting of the plurality ofillumination settings corresponds to a respective illumination emitterof the illumination source; and wherein each weight corresponds to anintensity of the respective illumination emitter.

In some embodiments, the classification model is configured to receivean input phase image or an input fluorescence image and output aclassification result indicative of one class out of a plurality ofpre-defined classes.

In some embodiments, the plurality of pre-defined classes comprises ahealthy class and a diseased class.

In some embodiments, the machine-learning model is a GAN modelcomprising an attention layer comprising the plurality of weights, adiscriminator, and a generator.

In some embodiments, the machine-learning model is a conditional GANmodel.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the plurality of neural networks comprises aplurality of U-Net neural networks.

In some embodiments, the discriminator is a PatchGAN neural network.

In some embodiments, training, using the trained classification model,the machine-learning model comprises: applying the plurality of weightsto a plurality of bright-field training images; aggregating theplurality of weighted bright-field training images into an aggregatedbright-field image; inputting the aggregated bright-field training imageinto the machine-learning model to obtain an enhanced training image anda generator loss; inputting the enhanced training image into the trainedclassifier to obtain a classifier loss; augmenting the generator lossbased on the classifier loss; and updating the plurality of weightsbased on the augmented generator loss.

In some embodiments, the method further comprises: classifying theenhanced image using the trained classifier.

In some embodiments, the method further comprises: displaying theenhanced image.

An exemplary system for generating enhanced images of biological samplescomprises: one or more processors; a memory; and one or more programs,wherein the one or more programs are stored in the memory and configuredto be executed by the one or more processors, the one or more programsincluding instructions for: obtaining, using a microscope, an image of abiological sample illuminated using an illumination pattern of anillumination source, wherein the illumination pattern is determined by:training a classification model configured to receive an input image andoutput a classification result, training, using the trainedclassification model, a machine-learning model having an plurality ofweights corresponding to a plurality of illumination settings, andidentifying the illumination pattern based on the plurality of weightsof the trained machine-learning model; and generating an enhanced imageof the biological sample by inputting the obtained image of thebiological sample into the trained machine-learning model.

An exemplary non-transitory computer-readable storage medium stores oneor more programs for generating enhanced images of biological samples,the one or more programs comprising instructions, which when executed byone or more processors of an electronic device, cause the electronicdevice to: obtain, using a microscope, an image of a biological sampleilluminated using an illumination pattern of an illumination source,wherein the illumination pattern is determined by: training aclassification model configured to receive an input image and output aclassification result, training, using the trained classification model,a machine-learning model having an plurality of weights corresponding toa plurality of illumination settings, and identifying the illuminationpattern based on the plurality of weights of the trainedmachine-learning model; and generate an enhanced image of the biologicalsample by inputting the obtained image of the biological sample into thetrained machine-learning model.

An exemplary method of evaluating a treatment with respect to a diseaseof interest comprises: receiving a first plurality of images depicting afirst set of healthy biological samples not affected by the disease ofinterest; receiving a second plurality of images depicting a second setof untreated biological samples affected by the disease of interest;receiving a third plurality of images depicting a third set of treatedbiological samples affected by the disease of interest and treated bythe treatment; inputting the first plurality of images into a trainedmachine-learning model to obtain a first plurality of enhanced images;inputting the second plurality of images into the trainedmachine-learning model to obtain a second plurality of enhanced images;inputting the third plurality of images into the trainedmachine-learning model to obtain a third plurality of enhanced images;comparing the first plurality of enhanced images, the second pluralityof enhanced images, and the third plurality of enhanced images toevaluate the treatment.

In some embodiments, the first plurality of images, the second pluralityof images, and the third plurality of images are bright-field images.

In some embodiments, the first plurality of enhanced images, the secondplurality of enhanced images, and the third plurality of enhanced imagesare fluorescence images.

In some embodiments, the first plurality of enhanced images, the secondplurality of enhanced images, and the third plurality of enhanced imagesare phase images.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment comprises: identifying, ineach image, a signal associated with a biomarker.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment further comprises: determininga first distribution based on signals of the biomarker in the firstplurality of enhanced images; determining a second distribution based onsignals of the biomarker in the second plurality of enhanced images; anddetermining a third distribution based on signals of the biomarker inthe third plurality of enhanced images.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment further comprises: comparingthe first distribution, the second distribution, and the thirddistribution to evaluate the treatment.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment comprises: determining, foreach image, a score indicative of the statement of the disease ofinterest.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment further comprises: determininga first distribution based on scores of the first plurality of enhancedimages; determining a second distribution based on scores of the secondplurality of enhanced images; and determining a third distribution basedon scores of the third plurality of enhanced images.

In some embodiments, comparing the first plurality of enhanced images,the second plurality of enhanced images, and the third plurality ofenhanced images to evaluate the treatment further comprises: comparingthe first distribution, the second distribution, and the thirddistribution to evaluate the treatment.

In some embodiments, the treatment is a first treatment, the methodfurther comprising: receiving a fourth plurality of images depicting afourth set of treated biological samples affected by the disease ofinterest and treated by a second treatment; inputting the fourthplurality of images into the trained machine-learning model to obtain afourth plurality of enhanced images; comparing the first plurality ofenhanced images, the second plurality of enhanced images, the thirdplurality of enhanced images, and the fourth plurality of enhancedimages to compare the first treatment and the second treatment.

In some embodiments, the method further comprises: selecting a treatmentout of the first treatment and the second treatment based on thecomparison.

In some embodiments, the method further comprises: administering theselected treatment.

In some embodiments, the method further comprises: providing a medicalrecommendation based on the selected treatment.

In some embodiments, the trained machine-learning model is is a GANmodel comprising a discriminator and a generator.

In some embodiments, the machine-learning model is a conditional GANmodel.

In some embodiments, the generator comprises a plurality of neuralnetworks corresponding to a plurality of frequency groups.

In some embodiments, each neural network of the plurality of neuralnetworks is configured to generate wavelet coefficients for a respectivefrequency group.

In some embodiments, the discriminator is a PatchGAN neural network.

An exemplary system for evaluating a treatment with respect to a diseaseof interest comprises: one or more processors; a memory; and one or moreprograms, wherein the one or more programs are stored in the memory andconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: receiving a first plurality ofimages depicting a first set of healthy biological samples not affectedby the disease of interest; receiving a second plurality of imagesdepicting a second set of untreated biological samples affected by thedisease of interest; receiving a third plurality of images depicting athird set of treated biological samples affected by the disease ofinterest and treated by the treatment; inputting the first plurality ofimages into a trained machine-learning model to obtain a first pluralityof enhanced images; inputting the second plurality of images into thetrained machine-learning model to obtain a second plurality of enhancedimages; inputting the third plurality of images into the trainedmachine-learning model to obtain a third plurality of enhanced images;comparing the first plurality of enhanced images, the second pluralityof enhanced images, and the third plurality of enhanced images toevaluate the treatment.

An exemplary non-transitory computer-readable storage medium stores oneor more programs for evaluating a treatment with respect to a disease ofinterest, the one or more programs comprising instructions, which whenexecuted by one or more processors of an electronic device, cause theelectronic device to: receiving a first plurality of images depicting afirst set of healthy biological samples not affected by the disease ofinterest; receiving a second plurality of images depicting a second setof untreated biological samples affected by the disease of interest;receiving a third plurality of images depicting a third set of treatedbiological samples affected by the disease of interest and treated bythe treatment; inputting the first plurality of images into a trainedmachine-learning model to obtain a first plurality of enhanced images;inputting the second plurality of images into the trainedmachine-learning model to obtain a second plurality of enhanced images;inputting the third plurality of images into the trainedmachine-learning model to obtain a third plurality of enhanced images;comparing the first plurality of enhanced images, the second pluralityof enhanced images, and the third plurality of enhanced images toevaluate the treatment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an exemplary process for training a machine-learningmodel configured to generate synthetic data (e.g., image data) at scalein a low-cost manner, in accordance with some embodiments.

FIG. 2A illustrates an exemplary process of training and applying themachine learning model, in accordance with some embodiments.

FIG. 2B illustrates an exemplary process of training and applying themachine learning model, in accordance with some embodiments.

FIG. 3A illustrates the training process of the machine-learning model,in accordance with some embodiments.

FIG. 3B illustrates operations of an exemplary generator, in accordancewith some embodiments.

FIG. 3C illustrates the training process of the machine-learning model,in accordance with some embodiments.

FIG. 3D illustrates the training process of the machine-learning model,in accordance with some embodiments.

FIG. 4 illustrates an exemplary image formation model, in accordancewith some embodiments.

FIG. 5 illustrates outputs of an exemplary trained generator using astatic microscope setup, in accordance with some embodiments.

FIG. 6A illustrates an exemplary process for determining the robustnessof the generated images, in accordance with some embodiments.

FIG. 6B illustrates an exemplary comparison between two classifiers, inaccordance with some embodiments.

FIG. 7A illustrates an exemplary method for training a machine-learningmodel to generate images of biological samples.

FIG. 7B illustrates an exemplary method for generating enhanced imagesof biological samples.

FIG. 8 illustrates an exemplary electronic device in accordance withsome embodiments.

FIG. 9 illustrates an exemplary optical system, in accordance with someembodiments.

FIG. 10 illustrates an exemplary method for analyzing images of abiological sample using a programmable SLM of an optical system, inaccordance with some embodiments.

FIG. 11 illustrates an exemplary method for training a machine-learningmodel, in accordance with some embodiments.

FIGS. 12A and 12B illustrate a side-by-side comparison of classificationresults of two classification models based on the same input images, inaccordance with some embodiments.

FIGS. 12C and 12D illustrate a side-by-side comparison of the sameclassification results in FIGS. 9A and 9B, respectively, with adifferent color scheme to show the two models' resistance to batcheffects, in accordance with some embodiments.

FIG. 12E illustrates the Euclidean distance metric corresponding to areal image set, a generated image set, and a truly batch-invariant imageset, in accordance with some embodiments.

FIG. 13A illustrates an exemplary generated image, in accordance withsome embodiments.

FIG. 13B illustrates an exemplary generated image, in accordance withsome embodiments.

FIG. 13C illustrates an exemplary generated image, in accordance withsome embodiments.

FIGS. 14A and 14B illustrate an exemplary process for training amachine-learning model (e.g., a GAN model) configured to generatesynthetic data (e.g., image data) and identifying an optimalillumination pattern for capturing input data for the machine-learningmodel, in accordance with some embodiments.

FIG. 15 illustrates a schematic diagram for training a machine-learningmodel, in accordance with some embodiments.

FIG. 16A illustrates synthetic images generated by an exemplary GANmodel to study the NASH disease, in accordance with some embodiments

FIG. 16B illustrates downstream analyses of the generated images, inaccordance with some embodiments.

FIG. 16C illustrates an exemplary process for evaluating a candidatetreatment, in accordance with some embodiments.

FIG. 16D illustrates an exemplary process for evaluating a candidatetreatment, in accordance with some embodiments.

FIG. 17A illustrates synthetic images generated by an exemplary GANmodel to study tuberous sclerosis (“TSC”), in accordance with someembodiments

FIG. 17B illustrates an exemplary process for evaluating a candidatetreatment, in accordance with some embodiments.

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure includes methods, systems, electronic devices,non-transitory storage media, and apparatuses for performing ML-basedgeneration of image data at scale. The generated image data (e.g., imagedata of biological samples) can provide sufficient richness and depthfor downstream processing (e.g., phenotyping). Further, embodiments ofthe present disclosure comprises a set of computational and hardwareoptimization methods that extends the current dimensionality of classicmicroscopy techniques.

Embodiments of the present disclosure can process multiple bright-fieldimages of biological samples and produce enhanced images of thebiological samples. The enhanced images include but are not limited to:fluorescence images, phase shift images, semantic map, polarization map,refractive map (2D and 3D), absorbance map, and other image modalities.Bright-field images of biological samples can be obtained at scale andat low cost due to inexpensive equipment (e.g., relative to fluorescencemicroscopes), ease of clinical deployment, and low processing andstorage resource requirements. Obtaining bright-field images isgenerally non-invasive and involves low photo-toxicity. Thus,bright-field images can be obtained efficiently and at scale. Theenhanced images provide sufficient richness and depth for downstreamprocessing (e.g., phenotypic exploration).

Embodiments of the present disclosure include a machine-learning modelthat is trained to receive a first type of image and translate the inputimage into other imaging modalities. An exemplary machine-learning modelcan receive the first type of image and translate the input image into asecond type of image (e.g., an enhanced image). In some embodiments,different image types refer to different imaging modalities. In someembodiments, the first type of images are bright-field images. Forexample, the bright-field images can be captured from illuminating invitro (or biopsy) cell samples with an inexpensive LED array. The secondtype of images include fluorescence images. The generated fluorescenceimages exhibit high contrast features that are not directly visible inbright-field images and can be used for downstream processing (e.g.,phenotyping).

Embodiments of the present disclosure reduce or eliminate the need tocapture real fluorescence images (or other special image modalities) ofbiological samples for downstream analysis, and allow bright-fieldimages to be widely used for a variety of purposes. This is particularlybeneficial to live cell imaging. For example, the disclosed methodscould be used for the monitoring and optimization of celldifferentiation experimental protocols. In the context of chemical orgenetic perturbation, time consuming activities linked to cells stainingand fixation could be avoided. In some embodiments, the dosing time,which is the incubation time of a drug with the cells under observationcould also be optimized by the software. Researchers would no longer toneed to arbitrarily decide the best incubation time, as the softwarewould be able to notify the researcher of the optimized incubation time.Further, the machine-learning techniques used to translate thebright-field images into other modalities require lower processing andstorage resource utilization. Thus, embodiments of the presentdisclosure present technical improvements in the field of medicalimaging while enhancing the operability and functioning of computingsystems.

Embodiments of the present disclosure further include a machine-learningmodel that is trained to receive the first type of image and translatethe input image into a third type of image. In some embodiments, thethird type of image include image data indicative of various opticalproperties (e.g., phase shift) of the biological sample captured.

Embodiments of the present disclosure further include a machine-learningmodel that is trained to receive the first type of image and translatethe input image into a fourth type of image. In some embodiments, thefourth type of image includes image data indicative of segmentation data(e.g., cell boundaries).

One of ordinary skill in the art should appreciate that embodiments ofthe present disclosure can further translate input images into numerousother types of image capturing a variety of imaging characteristics,such as a semantic map, a polarization map, a refractive map (2D or 3D),absorbance map, etc.

In some embodiments, a single machine-learning model is trained toperform multiple translation tasks simultaneously. For example, the samemachine-learning model can receive the first type of image and generatemultiple types of images (e.g., second type, third type, fourth type ofimage). The machine-learning model can be a Generative AdversarialNetwork (“GAN”) model. For example, the GAN network can be a conditionalGAN (“cGAN”) model.

In some embodiments, the machine-learning model converts an input imageinto its corresponding wavelet coefficients and generates one or moreoutput images in wavelet coefficients. The compact and multi-scalerepresentation of images coupled with the intrinsic non-linear optionsof a neural network achieves multiple goals at once. First, thewavelet-based representation solves the inherent class imbalance problemthat most of the generative models face. Specifically, most input imagedata comprise abundant low frequency signals but fewer high frequencysignals. Second, the wavelet-based representation extracts and maintainsthe overall geometry of the input image data. The discriminator of themodel ensures that the real and generated images (e.g., real v.generated fluorescence images) are indistinguishable. The generatedfluorescence images, or other enhanced image modality, corresponds tothe virtual staining of the sample. Because of the low photo-toxicity ofthe bright-field imaging modality, as well as its availability in theclinical setup, virtual staining by embodiments of the presentdisclosure can be performed on live cells as well as biopsy samples.

Embodiments of the present disclosure further includes hardwareoptimization methods. For example, embodiments of the present disclosurecan further optimize the illumination schema of a microscope (e.g., themicroscope that obtains the first type of images) dynamically. In someembodiments, the microscope that is used to capture the first type ofimages can be tuned or programmed to provide different illuminationschemes during the training process. During training of themachine-learning model, an optimal illumination scheme for capturing thefirst type of images can be identified. The optimal illumination schemecan be used to capture the first type of images (e.g., bright-fieldimages) so as to extract the best representation of the biologicalsample for wavelet-based image transformation (e.g., for downstreamphenotypic exploration).

Embodiments of the present disclosure further includes evaluating therobustness of the generated images of the machine-learning model. Insome embodiments, a first downstream classifier is trained using realimages (e.g., real fluorescence images), and a second downstreamclassifier is trained using generated images (e.g., generatedfluorescence images). The performance of the two classifiers can becompared to evaluate the robustness of the generated images as trainingdata in downstream tasks.

Thus, embodiments of the present disclosure comprise an integratedplatform that simultaneously solves many problems: image enhancement,phase retrieval, low photo-toxicity, realistic virtual painting ofbright-field images, and robustness in downstream tasks. Embodiments ofthe present disclosure can evaluate the robustness of the generatedimages via downstream classification tasks. These tasks are integratedto the platform and close the loop of data generation from non-invasivebright-field images to fluorescent images. For example, the system mayoptimize parameters of the bright-field microscope acquisition systemduring use. Illumination patterns of the LED array, and other parametersof the bright-field microscope acquisition system, including, forexample, the focus position of the microscope objective and activationtimings of a spatial light modulator (SLM) may be optimized byback-propagation during downstream classification tasks.

Further still, in some embodiments the platform performs a cascade ofperturbations to the cells and learn to optimize the illumination schemeto extract the best representation of the cells for phenotypicexploration.

Some embodiments of the present disclosure can identify one or moreoptimal illumination patterns for capturing image data. In someembodiments, an illumination pattern can indicate whether eachillumination emitter of an illumination source (e.g., each LED on a LEDarray) is to be turned on or off and the intensity of each illuminationemitter. The system can determine an optimal illumination pattern bytraining a machine-learning model having an attention layer comprisingan plurality of weights corresponding to the intensities of a pluralityof illumination emitters (e.g., a plurality of weights corresponding tothe intensities of a plurality of LEDs on the LED array). Duringtraining of the machine-learning model, the plurality of weights areapplied to a plurality of training images (e.g., bright-field images)illuminated by different illumination emitters. The aggregated image canbe inputted into the machine-learning model to determine a loss and themodel, including the weights in the attention layer, can be updatedbased on the loss accordingly. After training, an illumination patterncan be determined based on the weights in the attention layer of thetrained machine-learning model, as each weight can correspond to thedesired intensity level of the corresponding illumination emitter.Accordingly, the process involves only capturing images using a limitednumber of illumination settings (e.g., turning on a single illuminationemitter at a time to capture images) and does not require physicallyadjusting the intensities of the illumination emitters in order toidentify an optimal illumination pattern.

Some embodiments of the present disclosure can train a machine-learningmodel such that the synthetic image data generated by themachine-learning model can provide the same performance in downstreamanalyses as real images. In some embodiments, training themachine-learning model comprises first training a classifiercorresponding to the downstream task (e.g., classifying healthy v.diseased tissues based on an image) and then using the outputs of theclassifier to guide the training of the machine-learning model.

Some embodiments of the present disclosure can evaluate candidatetreatments with respect to a disease of interest. In some embodiments,the system receives a first plurality of images depicting a first set ofhealthy biological samples not affected by the disease of interest;receives a second plurality of images depicting a second set ofuntreated biological samples affected by the disease of interest; andreceives a third plurality of images depicting a third set of treatedbiological samples affected by the disease of interest and treated bythe candidate treatment. The images are inputted into a machine-learningmodel to obtain enhanced images, and the enhanced images are compared toevaluate the treatment (e.g., by analyzing the distributions of theimages).

The following description sets forth exemplary methods, parameters, andthe like. It should be recognized, however, that such description is notintended as a limitation on the scope of the present disclosure but isinstead provided as a description of exemplary embodiments.

In some embodiments, an exemplary optical system comprises aprogrammable spatial light modulator (“SLM”). The SLM of the opticalsystem can improve the performance of a machine-learning model via thetraining stage (e.g., by providing a rich training dataset) and/or viathe inference stage (e.g., by providing input data under a variety ofoptical settings or an optimal setting). The SLM is programmed withoutrequiring any mechanical movement or mechanical modifications to theoptical system.

The SLM provides additional degrees of freedom and sources of contrastto control the microscope in a programmable way. For example, the SLMcan be programmed to generate optical aberrations that enhance criticalphenotypes. As another example, the SLM can be programmed to providedifferent modulations, thus producing a variety of images that allow theexploration of deep samples at a high speed. The SLM also allowsidentification of an optimal imaging setup to infer cellular phenotypesand reconstruct alternative images modalities in a supervised fashion.Multi-focus acquisitions are possible without any mechanical movements,thus accelerating and improving the downstream tasks. Three-dimensionalphase tomography and reconstruction are therefore accelerated andimproved.

Although the following description uses terms “first,” “second,” etc. todescribe various elements, these elements should not be limited by theterms. These terms are only used to distinguish one element fromanother. For example, a first graphical representation could be termed asecond graphical representation, and, similarly, a second graphicalrepresentation could be termed a first graphical representation, withoutdeparting from the scope of the various described embodiments. The firstgraphical representation and the second graphical representation areboth graphical representations, but they are not the same graphicalrepresentation.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

The term “if” is, optionally, construed to mean “when” or “upon” or “inresponse to determining” or “in response to detecting,” depending on thecontext. Similarly, the phrase “if it is determined” or “if [a statedcondition or event] is detected” is, optionally, construed to mean “upondetermining” or “in response to determining” or “upon detecting [thestated condition or event]” or “in response to detecting [the statedcondition or event],” depending on the context.

FIG. 1 illustrates an exemplary process for training a machine-learningmodel configured to generate enhanced images of biological samples, inaccordance with some embodiments.

With reference to FIG. 1, training data 120 comprises a first type ofimages 122 and a second type of images 124. In some embodiments, theimages are images of biological samples, and the biological samples caninclude one or more collections of stained, unstained, perturbed and/orunperturbed biological samples.

In some embodiments, the first type of image data 122 comprises a set ofbright-field images, and the second type of image data 124 comprisesimages in a different modality (e.g., fluorescence images). The firsttype of images 122 can be obtained using a bright-field microscope,while the second type of images 124 can be obtained using a fluorescencemicroscope.

As shown in FIG. 1, the first type of images 122 are obtained based onone or more illumination pattern(s) 110. An illumination pattern isindicative of the settings with which an object is illuminated. In someembodiments, an illumination pattern can be defined by one or moreparameters indicating the spatial relationship (e.g., distance, angle)between the object and the illumination source, one or more parametersindicating the setup of the illumination source, or a combinationthereof. For example, an illumination pattern can indicate a set ofactivated LED light sources, a specific out-of-focus/polarization setup,etc.

In some embodiments, the microscope that captures the first type ofimages can be a microscope that supports multiple illumination patterns.For example, the microscope can provide a programmable illuminationsource (e.g., LED array, laser), an adaptive optics system (SLM,micro-mirrors), or a combination thereof. By updating the illuminationpattern (e.g., controlling illumination sources and/or the pupilfunction of the optics system), many representations of the biologicalsample corresponding to multiple illumination patterns can be acquired.

In some embodiments, the training data 120 can be organized asthree-dimensional image data (e.g., an image array). For example, thetraining data 120 can be of dimensions (B, C, H, W) in which B indicatesthe batch size, C indicates the number of channels (i.e., illuminationpatterns), H indicates the height, and W indicates the width. C equals 1if there is only a single bright-field image, and C is larger than 1 ifthere is a stack of bright-field images.

In some embodiments, one or more images in the training data 120 can benormalized before they are used to train the machine-learning model 100.For example, fluorescence images can be normalized based on illuminationor intensity parameters.

With reference to FIG. 1, the training data 120 is used to train themachine-learning model 100. In some embodiments, the model 100 is aGenerative Adversarial Network (“GAN”) model. For example, the GANnetwork can be a conditional GAN (“cGAN”) model. As described in detailbelow, the GAN network comprises a generator and a discriminator. Thegenerator is trained to receive the first type of images (e.g.,bright-field images) and translate the input images into the second typeof images (e.g., fluorescence images). The training step includescomparing real images of the first type 122 with real images of thesecond type 124 to determine a ground truth baseline for what awavelet-based transformation of the first type of image 122 should seekto accomplish. During training, the generator output (i.e., thegenerated fluorescence images) can be connected directly to thediscriminator input. The discriminator is trained to distinguish thegenerated images of the second type (e.g., generated fluorescenceimages) from real images of the second type (e.g., real fluorescenceimages). Through back-propagation, the discriminator's output can beused by the generator to update the generator's weights such that thegenerator learns to generate images that the discriminator will classifyas real images.

In some embodiments, the illumination parameters can be updated duringthe training of the model 100. Thus, during training of the model 100,the illumination scheme can be continually updated and training data canbe obtained according to the updated illumination scheme to furthertrain the model 100, as described in detail below.

FIGS. 2A and 2B illustrate exemplary processes 200 and 250 for trainingand applying the machine learning model (e.g., model 100), in accordancewith some embodiments. FIG. 2A illustrates the process when theparameters of the microscope cannot be changed during acquisition of thetraining data. FIG. 2B illustrates the process when the parameters ofthe microscope can be changed during acquisition of the training dataand thus an optimal illumination pattern can be identified.

Each of the processes can be performed at least in part using one ormore electronic devices. In some embodiments, the blocks of each processstep depicted in FIGS. 2A and 2B can be divided up between multipleelectronic devices. In each process, some blocks are, optionally,combined, the order of some blocks is, optionally, changed, and someblocks are, optionally, omitted. In some examples, additional steps maybe performed in combination with the process. Accordingly, theoperations as illustrated (and described in greater detail below) areexemplary by nature and, as such, should not be viewed as limiting.

In FIG. 2A, the parameters of the microscope cannot be changed duringthe acquisition of the bright-field images. Thus, all of thebright-field images in the training data (e.g., training data 120) areobtained based on the same illumination pattern.

At block 204, training data (e.g., training data 120 in FIG. 1) isobtained at least partially according to the default illuminationpattern. As described above with reference to FIG. 1, the training datacomprises a first type of images (e.g., bright-field images), which areobtained according to the default illumination pattern by a bright-fieldmicroscope, and a second type of images (e.g., fluorescence images)obtained by a fluorescence microscope.

At block 206, a machine-learning model (e.g., model 100 in FIG. 1) istrained based on the training data. The model can be a GAN model. Forexample, the GAN network can be a conditional GAN (“cGAN”) model.

FIGS. 3A, 3C, and 3D illustrates the training process of themachine-learning model, in accordance with some embodiments. The GANmodel comprises the generator 302 and a discriminator 304. The generator302 is trained to receive a first type of images 310 (e.g., bright-fieldimages) and generate a second type of images 312 (e.g., fluorescenceimages). In some embodiments, the first type of images 310 comprises thebright-field image array described above.

During training, the generator output (i.e., the generated fluorescenceimages) can be connected directly to the discriminator input. Thediscriminator 304 is trained to distinguish the generated images of thesecond type 312 (e.g., generated fluorescence images) from real imagesof the second type 314 (e.g., real fluorescence images). In someembodiments, the real images of the second type 314 comprises thefluorescence image array described above.

Through back-propagation, the discriminator's output can be used by thegenerator to update the generator's weights such that the generatorlearns to generate images that the discriminator will classify as realimages, as described in detail below. In some embodiments, the generator302 and the discriminator 304 are neural networks.

FIG. 3B illustrates operations of an exemplary generator, in accordancewith some embodiments. The input image 352 can be a single image from abright-field image array or an entire bright-field image array. Forexample, the input image 352 can be of dimensions (B, C, H, W) in whichB indicates the batch size, C indicates the number of channels, Hindicates the height, and W indicates the width. C equals 1 if there isonly a single bright-field image, and C is larger than 1 if there is astack of bright-field images.

With reference to FIG. 3B, the generator comprises a series ofconvolutional layers for down-sampling the input image 352 to obtain adown-sampled input image 354. In some embodiments, the input image 353can be down-sampled to half of its original size. For example, if theinput is 256*256 in the spatial size, it will be reduced to 128*128.

The down-sampled image 354 is then passed to a plurality of neuralnetworks. In the depicted example, the plurality of neural networkscomprises four U-Net neural networks. A U-Net network is a convolutionalnetwork for image-to-image translation. Details of the design andimplementation of a U-Net network can be found, for example, inRonneberger et al., “U-Net: Convolutional Networks for Biomedical ImageSegmentation,” which is hereby incorporated by reference in itsentirety.

The plurality of neural networks can correspond to different frequencygroups in the wavelet domain. In the depicted example, the four U-Netnetworks are responsible for low frequency, high frequency (horizontal),high frequency (vertical), and high frequency (diagonal), respectively.In signal processing low frequency signals correspond to very largefeatures with respect to the size of the image (for example when imagingcells, having a size magnitude of the order of the cytoplasm or thenucleus). High frequency information are very fine small image features(for example, having a size magnitude of the order of mitochondria,microtubules). Low frequency signals correspond to the first scale ofwavelet coefficients. The high frequency are encoded in higher scalewavelet coefficients. In some embodiments, the plurality of neuralnetworks operate independently and do not share weights.

As shown, three of the four U-Net branches correspond to high frequencyblocks. Low frequency information is relatively easy to recover; thus,having more computing power dedicated to the high frequency guaranteethe reconstruction of fine details in the images. Therefore, the lossfunction operating in the wavelet domain benefits from this organizationof the signal (3 times more high frequency information than lowfrequency information)

Each neural network is configured to output (or predict) waveletcoefficients for the respective frequency group. A loss function isapplied to the predicted wavelet coefficients 356 and the true waveletcoefficients of the real fluorescence image. The loss function isdescribed further below with reference to FIGS. 3C and 3D.

The generated fluorescence image 358 in the image domain can be obtainedby applying inverse wavelet transform on predicted coefficients 356.

FIG. 3C illustrates the back-propagation process of the discriminator304, according to some embodiments. The discriminator 304 is a model(e.g., a neural network) trained to provide an output based on a givenimage. The training data of the discriminator 304 comprises real imagesof the second type 314 (e.g., real fluorescence images) and syntheticimages of the second type 312 (e.g., generated fluorescence images)generated by the generator 302.

In some embodiments, the discriminator 304 is a PatchGAN network.Details of the design and implementation of the PatchGAN network can befound, for example, in Isola et al., “Image-to-Image Translation withConditional Adversarial Networks,” which is incorporated by reference inits entirety.

During training of the discriminator 304, a discriminator loss 322 canbe calculated based on the generator's outputs (i.e., the predictedwavelet coefficients). In some embodiments, the discriminator lossfunction is a Wasserstein discriminator loss and calculated as follows:

${\nabla_{w}\frac{1}{m}}{\sum_{i = 1}^{m}\left\lbrack {{f\left( x^{(i)} \right)} - {f\left( {G\left( z^{(i)} \right)} \right)}} \right.}$

where f(x) is the discriminator's output based on wavelet coefficientsof a real fluorescence image, w is the model weights of thediscriminator, m is the size of the mini-batch, f is the discriminatormodel, x is the real image, z is the input (bright-field), G is thegenerator model, and f(G(z)) is the discriminator's output based on thepredicted wavelet coefficients corresponding to a synthetic fluorescenceimage.

The discriminator 304 is configured to maximize this function. In otherwords, it tries to maximize the difference between its output based onreal images and its output based on synthetic images. As depicted inFIG. 3C, the discriminator updates its weights through back-propagationbased on the discriminator loss 332 through the discriminator network.

FIG. 3D illustrates the back-propagation process of the generator 302,according to some embodiments. The generator 302 is a neural networkconfigured to receive a first type of images 310 and generate a secondtype of images 312, as described with reference to FIGS. 3A and 3B. Thepredicted wavelet coefficients by the generator are inputted into thediscriminator 304. Depending on the discriminator's outputs, a generatorloss 324 can be calculated. In some embodiments, the generator lossfunction is a Wasserstein generator loss and calculated as follows:

${\nabla_{\theta}\frac{1}{m}}{\sum_{i = 1}^{m}\left\lbrack {f\left( {G\left( z^{(i)} \right)} \right)} \right.}$

where f(x) is the discriminator's output based on wavelet coefficientsof a real fluorescence image, m is the size of the mini-batch, f is thediscriminator model, z is the input (bright-field), G is the generatormodel, and f(G(z)) is the discriminator's output based on the predictedwavelet coefficients.

The reconstruction loss, operating in the wavelet domain, has theproperty to naturally balance the contribution of low and highfrequencies. As shown in FIG. 3B, the wavelet coefficients can be splitinto two categories: low (one block) and high frequency (3 blocks).Three of the four UNet branches can be dedicated to the high frequencyblocks. Low frequency information can more easily be recovered;therefore, having more computing power dedicated to the high frequencyhelps with the reconstruction of fine details in the images. The lossfunction operating in the wavelet domain, benefits from thisorganization of the signal (three times more high frequency informationthan low frequency information). In some embodiments, the waveletcoefficients can be split into more than two categories, for example,three categories (high, medium, low frequencies), four categories, ormore than four categories.

The generator 302 is configured to maximize this function. In otherwords, it tries to maximize the discriminator's output based on itssynthetic images. In some embodiments, the generator loss isback-propagated through both the discriminator 304 and the generator 302to obtain gradients, which are in turn used to adjust the generatorweights.

In some embodiments, the generator 302 and the discriminator 304 aretrained in alternating periods. In each period, the discriminator trainsfor one or more epochs, and the generator trains for one or more epochs.During the discriminator training, the generator may remain constant.Similarly, during the generator training, the discriminator may remainconstant.

In some embodiments, the generator can translate the input image into athird type of image (e.g., phase shift image). For example, in additionto the generated fluorescence images, the generator may also outputphase shift images in which each pixel indicates local value of thephase in the image (e.g., −5 to 5 phase information) that can beconverted. A physics-based image formation model can be used to generatereal phase shift images (i.e., the ground-truth phase shift images). Theimage formation model generates images given an absolute knowledge ofthe microscope (e.g., the aberrations of the optical system) as well asthe optical properties of the sample captured (e.g., refractive index,phase). Because the optical properties of the sample can be comparedbetween samples, the risk of batch effects in downstream tasks is almostnull. A physics based model allows the incorporation of solid a prioriknowledge in the generative process.

With reference to FIG. 4, during training, S, the illumination source,is optimized in order to retrieve X with high fidelity. The microscopePSF (or pupil function) can also be optimized with a spatial lightmodulator or a set of micro-mirrors. Additionally, the polarization of Scan also be modulated in order to inject more contrast in the collectedimages.

I=∫|∫X(r)e ^(fr) P(f−r)dr| ² S(f)df

In the formula above, S(f) refers to the partially coherent illuminationsource (LED array). X(r) refers to the sample's complex electronicfield. P(r) refers to the microscope's point spread function (PSF).RI(r) refers to the refractive index. I refers to the image. The forwardmodel is applied outside of the training loop to obtain the ground truthphase.

In some embodiments, the generator can translate the input image into afourth type of image. In some embodiments, the fourth type of imageinclude image data indicative of segmentation data (e.g., cellboundaries). In some embodiments, the loss function depends on the imagemodality supported. For semantic segmentation, a L1 norm can be used toinfer the discrete labels in the images. For another image modality,another branch can be added to the generator to output the new modality.

Turning back to FIG. 2A, at block 208, metadata can be associated withthe trained machine-learning model. The metadata can include the defaultillumination pattern (e.g., parameters of the bright-field microscope).

At block 210, one or more images of the first type (e.g., bright-fieldimages) are obtained. In some embodiments, the images are obtained usingthe same illumination pattern (e.g., as indicated in the metadata inblock 208).

At block 212, the one or more images are inputted into the generator(e.g., generator 302) of the trained machine-learning model. Thegenerator has been trained to translate the first type of images intothe second type of images (e.g., fluorescence images). At block 214, oneor more images of the second type are obtained. As described below, thegenerated fluorescence images in turn can be used as training data forother machine-learning models, thus eliminating the need to obtain realfluorescence images as training data.

FIG. 2B illustrate the process when the parameters of the microscope canbe changed during acquisition of the training data and thus an optimalillumination pattern can be identified. For example, the parameters ofthe microscope can be programmable such that multiple illuminationpatterns can be provided by the microscope.

At block 252, an illumination pattern is loaded onto a microscope. Atblock 254, bright-field training images are captured according to theillumination pattern (e.g., bright-field images) and fluorescencetraining images are also captured by a fluorescence microscope. Blocks252 and 254 can be repeatedly performed, as indicated by the arrow 253.In other words, a sequence of illumination patterns can be loaded ontothe microscope and training data corresponding to the sequence ofillumination patterns can be obtained. The sequence of illuminationpatterns is also referred to an illumination scheme herein.

At block 256, a machine-learning model is trained based on the trainingdata. The model can be a GAN model that operates as described withreference to FIGS. 3A-D. For example, the GAN network can be aconditional GAN (“cGAN”) model.

During training, the model updates the illumination pattern iteratively.The illumination pattern is updated by back-propagating the gradient ofthe loss (e.g., generator loss) to the parameters of the microscope. Thetraining procedure of the model minimizes the overall imaging time andminimizes the loss functions associated with translation orclassification tasks.

During training, the system determines which illumination pattern leadsto the smallest loss (e.g., generator loss). In some embodiments, themodel produces a first generator loss when translating a first image,produces a second generator loss when translating a second image, etc.The losses can be compared and the smallest loss can be identified. Atblock 260, the illumination pattern that has produced the smallest losscan be identified and a new illumination scheme can be identifiedaccordingly. For example, the new illumination scheme can include thebest illumination pattern and/or one or more new illumination patternssimilar to the best illumination pattern. The new illumination schemecan also exclude previously included illumination patterns that haveresulted in the biggest loss. As indicated by arrow 258, the identifiedillumination scheme can be loaded to the microscope to obtain additionaltraining data. This process can be repeated until no more improvement(e.g., on the generator loss) is observed. The optimal illuminationscheme can be stored at block 261.

At block 262, one or more images of the first type (e.g., bright-fieldimages) are obtained. In some embodiments, the images are obtained usingthe optimal illumination scheme stored in block 261.

At block 264, the one or more images are inputted into the generator(e.g., generator 302) of the trained machine-learning model. Thegenerator has been trained to translate the first type of images intothe second type of images (e.g., fluorescence images). At block 266, oneor more images of the second type are obtained. As described below, thegenerated fluorescence images in turn can be used as training data forother machine-learning models, thus eliminating the need to obtain realfluorescence images as training data.

FIG. 5 illustrates outputs of an exemplary trained generator using astatic microscope setup. The trained generator receives inputbright-field images and generates phase images and fluorescence images504. For comparison, FIG. 5 also shows the corresponding real phaseimages and fluorescence images 506. In FIG. 5, each input image is acomposite of four different illuminations. In some embodiments, theinput image is a false color image representing the differentillumination patterns.

FIG. 6A illustrates an exemplary process for determining the robustnessof the generated images, in accordance with some embodiments. Twoclassifiers 612 and 614 are trained to receive a fluorescence image andprovide an output (e.g., a phenotype classification result). Classifier614 is trained based on real fluorescence images 604 (e.g., realfluorescence images), while classifier 612 is trained based on generatedfluorescence images 610. The generated fluorescence images are generatedby a trained generator 608 (e.g., generator 102, 302) based onbright-field images 506.

In some embodiments, classifier 612 is validated using generated images,while classifier 614 is validated using real images.

The performance of the classifiers 612 and 614 can be compared todetermine the robustness of the generated images. FIG. 6B illustrates anexemplary comparison between two such classifiers. In this particularsetup, the dataset is made of fluorescence images and 3D bright-fieldimages of primary hepatocytes from 12 healthy donors and 12 NASH donors.A GAN model is trained to generate fluorescence images from a stack ofout of focus bright-field images.

A comparison is made to determine whether a classifier trained ongenerated images and validated on real images performs equally well whentrained on real images and validated on generated images. As shown inFIG. 6B, there is no drop in the classification accuracy and moreimportantly, the geometry of the embeddings space of the real images isconserved on the generated images.

Some embodiments include a back-propagation module 616. Back-propagationmodule 616 can be used, for example, to improve the image acquisitionparameters used for acquiring bright-field images that are to beenhanced. For example, Xi can include a set parameters for theillumination patterns applied to a LED illumination array of abright-field microscope. Xi can also include imaging parameters relatedto the focus position of the microscope objective and or activationparameters of a spatial light modulator (SLM). All elements of Xi may bevariables of the optimization procedure. Therefore in 616 the gradientof the loss function estimated on downstream task [612] (e.gclassification, image to image translation, regression) may beback-propagated to optimize the parameters, Xi. For each update to thevalues of Xi a new set of images may be acquired generating a new dataset of images.

In some embodiments, the data used to train the generator 608, the dataused to train the first classifier 614 or the second classifier 612, andthe data used to evaluate the performance of the classifiers can includeoverlapping images. Images used in any of the above-described processescan be annotated based on the biological samples captured (e.g., type ofcells, whether diseased or healthy) and the perturbations. Theseannotations can be used for downstream classification tasks (e.g., astraining data to train classifiers 612 and 614 in FIG. 6A) andvalidation of the predictive models (e.g., to evaluate the performanceof the trained classifiers 612 and 614).

FIG. 7A illustrates an example of a method 700 for training amachine-learning model to generate images of biological samples inaccordance with one embodiment. The method includes obtaining aplurality of training images comprising a training image of a firsttype, and a training image of a second type at 702. The method furtherincludes generating, based on the training image of the first type, aplurality of wavelet coefficients using the machine-learning model at704; generating, based on the plurality of wavelet coefficients, asynthetic image of the second type at 706; comparing the synthetic imageof the second type with the training image of the second type at 708;and updating the machine-learning model based on the comparison at 710.

FIG. 7B illustrates an example of a method 750 for generating enhancedimages of biological samples in accordance with one embodiment. Themethod includes obtaining, using a microscope, an image of a biologicalsample at 752. The method also includes at 754 generating, based on theimage, an enhanced image of the biological sample using themachine-learning model of FIG. 7A.

FIG. 8 illustrates an example of a computing device in accordance withone embodiment. Device 800 can be a host computer connected to anetwork. Device 800 can be a client computer or a server. As shown inFIG. 8, device 800 can be any suitable type of microprocessor-baseddevice, such as a personal computer, workstation, server or handheldcomputing device (portable electronic device) such as a phone or tablet.The device can include, for example, one or more of processor 810, inputdevice 820, output device 830, storage 840, and communication device860. Input device 820 and output device 830 can generally correspond tothose described above, and can either be connectable or integrated withthe computer.

Input device 820 can be any suitable device that provides input, such asa touch screen, keyboard or keypad, mouse, or voice-recognition device.Output device 830 can be any suitable device that provides output, suchas a touch screen, haptics device, or speaker.

Storage 840 can be any suitable device that provides storage, such as anelectrical, magnetic or optical memory including a RAM, cache, harddrive, or removable storage disk. Communication device 860 can includeany suitable device capable of transmitting and receiving signals over anetwork, such as a network interface chip or device. The components ofthe computer can be connected in any suitable manner, such as via aphysical bus or wirelessly.

Software 850, which can be stored in storage 840 and executed byprocessor 810, can include, for example, the programming that embodiesthe functionality of the present disclosure (e.g., as embodied in thedevices as described above).

Software 850 can also be stored and/or transported within anynon-transitory computer-readable storage medium for use by or inconnection with an instruction execution system, apparatus, or device,such as those described above, that can fetch instructions associatedwith the software from the instruction execution system, apparatus, ordevice and execute the instructions. In the context of this disclosure,a computer-readable storage medium can be any medium, such as storage840, that can contain or store programming for use by or in connectionwith an instruction execution system, apparatus, or device.

Software 850 can also be propagated within any transport medium for useby or in connection with an instruction execution system, apparatus, ordevice, such as those described above, that can fetch instructionsassociated with the software from the instruction execution system,apparatus, or device and execute the instructions. In the context ofthis disclosure, a transport medium can be any medium that cancommunicate, propagate or transport programming for use by or inconnection with an instruction execution system, apparatus, or device.The transport readable medium can include, but is not limited to, anelectronic, magnetic, optical, electromagnetic or infrared wired orwireless propagation medium.

Device 800 may be connected to a network, which can be any suitable typeof interconnected communication system. The network can implement anysuitable communications protocol and can be secured by any suitablesecurity protocol. The network can comprise network links of anysuitable arrangement that can implement the transmission and receptionof network signals, such as wireless network connections, T1 or T3lines, cable networks, DSL, or telephone lines.

Device 800 can implement any operating system suitable for operating onthe network. Software 850 can be written in any suitable programminglanguage, such as C, C++, Java or Python. In various embodiments,application software embodying the functionality of the presentdisclosure can be deployed in different configurations, such as in aclient/server arrangement or through a Web browser as a Web-basedapplication or Web service, for example.

Optimization Using Spatial Light Modulators

In some embodiments, an exemplary optical system comprises aprogrammable SLM. An exemplary SLM comprise a high-resolution liquidcrystal panel with micron-sized individually addressable pixels, whichcan be used to shape the wavefront of an optical beam. Grey-level valueson the panel are converted into phase shifts. SLM can be used asprogrammable Fourier filters for generating contrast enhancement or asprogrammable diffractive optical elements for quantitative phasemicroscopy, in some embodiments.

The SLM can be programmed to generate various images during the trainingstage and/or during the inference stage of machine-learning models toimprove the performance of the machine-learning models, as describedherein.

During the inference stage of a trained machine-learning model, the SLMcan be programmed to generate different input images for the trainedmachine-learning model. For example, the SLM can be programmed togenerate input images for a trained image transformation model to obtainenhanced versions of the input images. The enhanced images can in turnbe used for downstream operations. As another example, the SLM can beprogrammed to generate input images for a trained classification modelto obtain more accurate classification results.

Further, the SLM can be programmed to generate different images astraining data during the training stage of a machine-learning model.Further, SLM can be iteratively programmed, during either the trainingstage or the inference stage, to identify an optimal setting forcapturing images that leads to the best performance of a givenmachine-learning model.

Accordingly, the SLM of the optical system can improve the performanceof a machine-learning model via the training stage (e.g., by providing arich training dataset) and/or via the inference stage (e.g., byproviding input data under a variety of settings or the optimalsetting). The SLM is programmed without requiring any mechanicalmovement or modifications to the optical system (e.g., a microscope).The SLM provides additional degrees of freedom to control themicroscope. Multi focus acquisitions are possible without any mechanicalmovements, thus accelerating and improving the downstream tasks in anefficient manner.

FIG. 9 illustrates an exemplary optical system, in accordance with someembodiments. An exemplary optical system comprises a light source 902(e.g., an LED array), objective lens 904, an SLM 906, and a camera 908.The optical system has a reflection mode. In the reflection mode, thelight source 902 provides illumination to a biological sample 910, whichgenerates reflected light. The reflected light travels through theobjective lens 904 and is captured by the camera 908.

With reference to FIG. 9, the dash line 912 indicates the intermediateimage plane. As shown, the SLM is placed in the imaging path between thebiological sample 910 and the camera 908. The SLM is configured toimpose spatially varying modulations on the reflected light. Forexample, the SLM allows shaping of an alternative Fourier plane beforethe reflected light is focused on the camera 908.

In some embodiments, both the light source 902 and the SLM 906 can beprogrammable, thus allowing additional degrees of freedom to control andoptimize the optical system without mechanically moving components ofthe optical system.

The configuration of the optical system in FIG. 9 is merely exemplary.One of ordinary skill in the art would appreciate that otherconfigurations of the optical system can be used to place the SLM in theimaging path of the optical system to apply optimization techniquesdescribed herein.

FIG. 10 illustrates an exemplary method for analyzing images of abiological sample using a programmable SLM of an optical system, inaccordance with some embodiments. Process 1000 is performed, forexample, using one or more electronic devices implementing a softwareplatform. In some examples, process 1000 is performed using aclient-server system, and the blocks of process 1000 are divided up inany manner between the server and one or more client devices. In otherexamples, process 1000 is performed using only one or more clientdevices. In process 1000, some blocks are, optionally, combined, theorder of some blocks is, optionally, changed, and some blocks are,optionally, omitted. In some examples, additional steps may be performedin combination with the process 1000. Accordingly, the operations asillustrated (and described in greater detail below) are exemplary bynature and, as such, should not be viewed as limiting.

At block 1002, an exemplary system (e.g., one or more electronicdevices) obtains a plurality of images of the biological sample (e.g.,sample 910 in FIG. 9). The plurality of images are generated using aplurality of configurations of an SLM of an optical system (e.g., SLM906 in FIG. 9). The SLM is located in an optical path between thebiological sample and an image recording device (e.g., camera 908).

In some embodiments, at least one configuration of the SLM is togenerate one or more optical aberrations in the resulting image(s). Theoptical aberrations can include spherical aberration, astigmatism,defocus, tilt, or any combination thereof. In some embodiments, moreinformation can be captured in and/or derived from images with opticalaberrations. As an example, astigmatism allows collection of multi focusplane information in one image. As another example, defocus allows thesystem to scan a sample without any mechanical movements. An exemplarymethod of generating optical aberrations is provided in “QuantitativePhase Imaging and Complex Field Reconstruction by Pupil ModulationDifferential Phase Contrast” by Lu et al., the content of which isincorporated by reference herein in its entirety.

In some embodiments, at least one configuration of the SLM is to enhanceone or more features. The one or more features can comprise a cellborder, an actin filament, nuclear shape, cytoplasm segmentation, or anycombination thereof. In some embodiments, the system providesenhancement of neurites with specific convolution kernel encoded in thetransfer function of the microscope. In some embodiments, the systemprovides enhancement of the response function of the microscope forcertain liquid phase separated biological objects (protein, RNA, lipid).This enhancement would provide the capability to detect and characterizethe composition of these objects. For example, using a helical phasepattern as phase filter in a Fourier plane gives rise to a doughnut-likepoint spread function. Convolution with an extended amplitude orphase-object leads to strong isotropic edge enhancement in the image. Ina homogeneous (region of the) sample, destructive interference occursbecause of the 7E phase shift across the doughnut (for any angle alongthe ring). Structure in the sample can give rise to less imperfectcancellation and thus to a local brightening in the image. Consequentlythe light is redistributed into edges and borderlines within the sample.Exemplary methods of enhancing features are provided in “Spiral phasecontrast imaging in microscopy” by Fürhapter et al., “Shadow effects inspiral phase contrast microscopy” by Fürhapter et al., “Quantitativeimaging of complex samples in spiral phase contrast microscopy” byBernet et al., “Upgrading a microscope with a spiral phase plate” byMaurer et al., the content of which is incorporated by reference hereinin their entirety.

In some embodiments, at least one configuration of the SLM is to reduceoptical aberrations. For example, for live imaging and continuousmonitoring of the cells, it is critical to reduce variability of thesamples coming from the plates or debris in the wells. The SLM can allowthese aberrations to be corrected. In some embodiments, the SLM can beused for iterative corrections of phase aberrations, e.g. based theGerchberg-Saxton algorithm. For example, the dark center of a singleoptical vortex cam be used as a critical sensor for residual phaseaberrations. Exemplary methods of enhancing features are provided in“Wavefront correction of spatial light modulators using an opticalvortex image” by Jesacher et al. and “Phase contrast microscopy withfull numerical aperture illumination” by Maurer et al., the content ofwhich is incorporated by reference herein in their entirety.

In some embodiments, the plurality of SLM configurations is to obtainimages of the biological sample at different depths. For example, theSLM allows for flexible image multiplexing, e.g., to combine images fromdifferent depths of the sample or for different settings of imagingparameters in one recorded image. Image multiplexing can facilitatequantitative phase microscopy. Imaging live sample has to be as fast aspossible to minimize stress on the cells. Mechanical movements for 3Dscanning of the samples are prohibitive. This system allows Fourierptychography to be performed and reconstruction of large 3D volumes outof low resolution images of a whole well. Exemplary methods ofmultiplexing are provided in “Depth-of-field-multiplexing in microscopy”by Maurer et al., “Differential interference contrast imaging using aspatial light modulator” by McIntyre et al., and “Quantitative SLM-baseddifferential interference contrast Imaging” by McIntyre et al., thecontent of which is incorporated by reference herein in their entirety.

In some embodiments, the system allows reconstruction of high-resolutionimages through optimized Fourier ptychography. Fourier ptychography is acomputational imaging technique based on optical microscopy thatincludes the synthesis of a wider numerical aperture from a set offull-field images acquired using different optical settings, resultingin increased resolution compared to a conventional microscope. Theimages in an image set can be acquired using different configurations ofthe LED and/or the SLM; the acquired image set can then be combinedusing an iterative phase retrieval algorithm into a finalhigh-resolution image that can contain up to a billion pixels (agigapixel) with diffraction-limited resolution, resulting in a highspace-bandwidth product.

A light source of the optical system can also be programmed. In someembodiments, the plurality of images at block 1002 are obtained using aplurality of configurations of the light source (e.g., light source 902)of the optical system. Exemplary methods for programming the lightsource are described herein, for example, with reference to FIGS. 1 and2B.

Turning back to FIG. 10, at block 1004, the system inputs the pluralityof images of the biological sample into a trained machine-learning modelto obtain one or more outputs.

In some embodiments, the trained model is an image transformation modelconfigured to generate, based on an input image of a first type, anoutput image of a second type (e.g., enhanced versions of the inputimages). In some embodiments, the enhanced versions of the input imagescomprise enhanced cellular phenotypes. In some embodiments, the trainedmodel is a GAN model or a self-supervised model. For example, thetrained model can be model 100 configured to receive bright-field imagesand generate fluorescence images.

In some embodiments, the trained model is a classification modelconfigured to provide a classification output. For example, the modelcan receive an input image and detect one or more pre-defined objects inthe input image, such as a diseased tissue.

FIG. 11 illustrates an exemplary method for training a machine-learningmodel, in accordance with some embodiments. Process 1100 is performed,for example, using one or more electronic devices implementing asoftware platform. In some examples, process 1100 is performed using aclient-server system, and the blocks of process 1100 are divided up inany manner between the server and one or more client devices. In otherexamples, process 1100 is performed using only one or more clientdevices. In process 1100, some blocks are, optionally, combined, theorder of some blocks is, optionally, changed, and some blocks are,optionally, omitted. In some examples, additional steps may be performedin combination with the process 1100. Accordingly, the operations asillustrated (and described in greater detail below) are exemplary bynature and, as such, should not be viewed as limiting.

At block 1102, an exemplary system (e.g., one or more electronicdevices) obtains a plurality of images of a biological sample. Theplurality of images are generated using a plurality of configurations ofan SLM of an optical system (e.g., SLM 906 in FIG. 9). The SLM islocated in an optical path between the biological sample and an imagerecording device (e.g., camera 908).

In some embodiments, the SLM configurations are configured to createdesirable effects in the resulting images. As discussed above, someconfigurations of the plurality of configurations of the SLM can be togenerate one or more optical aberrations (e.g., a spherical aberration,astigmatism, defocus, distortion, tilt, or any combination thereof), toenhance one or more features (e.g., a cell border, an actin filament,nuclear shape, cytoplasm segmentation, or any combination thereof), toreduce optical aberrations, to obtain images of the biological sample atdifferent depths, etc.

At block 1104, the system trains the machine-learning model using theplurality of images. In some embodiments, the trained model is an imagetransformation model configured to generate, based on an input image ofa first type, an output image of a second type (e.g., enhanced versionsof the input images). In some embodiments, the enhanced versions of theinput images comprise enhanced cellular phenotypes. In some embodiments,the trained model is a GAN model or a self-supervised model. Forexample, the trained model can be model 100 configured to receivebright-field images and generate fluorescence images. Exemplary methodsof training the model are described herein with reference to FIGS. 2A-Band 3A-3D.

In some embodiments, the trained model is a classification modelconfigured to provide a classification output. For example, the modelcan receive an input image and detect one or more pre-defined objects inthe input image, such as a diseased tissue. Training of theclassification model can be performed using the plurality of images andassociated labels.

In some embodiments, at block 1104, the SLM can be iterativelyprogrammed to identify an optimal SLM configuration for capturing imagesthat lead to an improved performance of a given machine-learning model.Specifically, at block 1106, the system trains the machine-learningmodel using a first image, where the first image is obtained using afirst configuration of the SLM of the optical system. At block 1108, thesystem evaluates the trained machine-learning model. At block 1110, thesystem, based on the evaluation, identifying a second configuration ofthe SLM. At block 1112, the system trains the machine-learning modelusing a second image, wherein the second image is obtained using thesecond configuration of the SLM of the optical system.

For example, at block 1106, the system trains the model using imagescorresponding to a first set of SLM configurations. Each image resultsin a corresponding loss based on a loss function of the model. At block1108, the system determines which SLM configuration of the first set ofSLM configurations results in the smallest loss (e.g., generator loss).At block 1110, the SLM configuration that has produced the smallest losscan be identified and a new second set of SLM configurations can beidentified accordingly. For example, the new set of SLM configurationscan include the best SLM configuration (i.e., the configuration thatproduced the smallest loss) from the first set and/or one or more newSLM configurations similar to the best SLM configuration. The new set ofSLM configurations can also exclude SLM configurations that haveresulted in the biggest loss from the first set. The new set of SLMconfigurations can be loaded onto the optical system to obtainadditional training data. This steps can be repeated until a thresholdis met, for example, when no more improvement (e.g., on the generatorloss) is observed. The optimal SLM configuration can be stored and usedto obtain input images.

While steps 1106-1112 are described as part of a training process, theycan be performed in other stages of the pipeline (e.g., inference stage)to identify the optimal SLM configuration for generating input images.In some embodiments, the light source and the SLM of the optical systemcan be iteratively programmed together to identify the best combinationof illumination pattern and SLM configuration for generating inputimages.

While FIGS. 9-11 describes optimization techniques using an SLMcomponent of an optical system, it should be appreciated the SLM can bereplaced with another hardware component that can alter the opticalfunction (e.g., pupil function) of the system, such as micro-mirrors,without departing from the spirit of the invention.

FIGS. 12A and 12B illustrate a side-by-side comparison of classificationresults of two classification models, in accordance with someembodiments. The classification model in FIG. 12A is trained using realimages captured by a microscope (e.g., real fluorescence images), whilethe classification model in FIG. 12B is trained using synthetic imagesgenerated using the techniques described herein (e.g., fluorescenceimages generated from bright-field images). The classification modelsdetermines which chemical compound that tissues depicted in an inputimage have responded to. Specifically, each model is configured toreceive an input image and output a classification result indicative ofone of 150 predefined chemical compounds. In the depicted example, eachof FIGS. 12A and 12B shows a Uniform Manifold Approximation andProjection (UMAP) in which each input image is represented as a point inthe UMAP. The color of the point represents the chemical compound theimage is classified as by the classification model. In some embodiments,the input images into the model in FIG. 12A are real images, while theinput images into the model in FIG. 12B are generated images.

FIGS. 12C and 12D illustrate a side-by-side comparison of the sameclassification results in FIGS. 12A and 12B, respectively, with adifferent color scheme to demonstrate the two models' resistance tobatch effects. Batch effects refer to situations where subsets (i.e.,batches) of data significantly differ in distribution due to irrelevant,instrument-related factors. Batch effects are undesirable because theyintroduce systematic errors, which may cause downstream statisticalanalysis to produce spurious results and/or obfuscate the signal ofinterest. In each of FIGS. 12C and 12D, the input images belong to threedifferent batches (e.g., from different plates or experiments), asindicated by different grey levels. As shown, FIG. 12D shows a greateroverlap among the points corresponding to the three batches, indicatingthat the generated images are more resistance to batch effects. FIG. 12Eillustrates the Euclidean distance metric corresponding to a real imageset (e.g., input images as shown in FIGS. 12A and 12C), a generatedimage set (e.g., input images as shown in FIGS. 12B and 12D), and atruly batch-invariant image set. The Euclidean distance metric for eachimage set measures the Euclidean distance between the image embeddingsfrom two separate batches (e.g., Real Image Batch 1 and Real Image Batch3 in FIG. 12C, Generated Image Batch 1 and Generated Image Batch 3 inFIG. 12D). For a truly batch-invariant image set, the mean score shouldbe 0 (i.e., no batch effects). As shown, the mean score for thegenerated image set is lower than the mean score for the real images,thus demonstrating superior batch invariance.

FIGS. 13A and 13B illustrate an exemplary generated phase image and anexemplary generated fluorescence image, in accordance with someembodiments. FIG. 13A is a phase image generated by a GAN modeldescribed herein, for example, from a bright-field image. FIG. 13B is afluorescence/bodipy image generated by the GAN from the samebright-field image. The bodipy image includes a virtual green hue tohighlight the presence of a biomarker. FIG. 13C shows FIG. 13B overlaidonto the phase image in FIG. 13A. Thus, the GAN model can be used fordisease modeling. For example, multiple samples can be obtained andimaged from a subject. The resulting series of bright-field images canbe analyzed by the GAN model to generate synthetic phase andfluorescence images to study disease perturbations.

FIGS. 14A-B illustrate an exemplary process for training amachine-learning model (e.g., a GAN model) configured to generatesynthetic data (e.g., image data) and identifying an optimalillumination scheme for obtaining input data for the machine-learningmodel, in accordance with some embodiments. Process 1400 is performed,for example, using one or more electronic devices implementing asoftware platform. In some examples, process 1400 is performed using aclient-server system, and the blocks of process 1400 are divided up inany manner between the server and one or more client devices. In otherexamples, process 1400 is performed using only one or more clientdevices. In process 1400, some blocks are, optionally, combined, theorder of some blocks is, optionally, changed, and some blocks are,optionally, omitted. In some examples, additional steps may be performedin combination with the process 1400. Accordingly, the operations asillustrated (and described in greater detail below) are exemplary bynature and, as such, should not be viewed as limiting.

At block 1402, an exemplary system (e.g., one or more electronicdevices) receives a plurality of training images. The plurality oftraining images are real images of biological samples and are alsoreferred to as the ground truth data. The plurality of training imagescomprises the type of image data that the GAN model is configured toreceive and the type(s) of image data that the GAN model is configuredto output. For example, if the GAN model is configured to receive abright-field image and output a fluorescence image and a phase image,the received plurality of images would include a plurality ofbright-field training images 1402 a (i.e., a GAN input data type), aplurality of fluorescence training images 1402 b (i.e., a GAN outputdata type), and a plurality of phase training images (i.e., a GAN outputdata type).

The bright-field training images in the plurality 1402 a can be capturedby illuminating in vitro (or biopsy) cell samples with an inexpensiveLED array using different illumination settings. The fluorescencetraining images in the plurality 1402 b can be captured after a dye isapplied to the biological sample (e.g., to enhance the visibility of abiomarker). The phase training images can be obtained using physics- oroptics-based models. It should be appreciated by one of ordinary skillin the art that the plurality of training images used in the process1400 can differ depending on the type of the image data that the GAN isconfigured to receive and output. In some embodiments, the plurality oftraining images comprises paired image data. For example, a bright-fieldimage, a fluorescence image, and a phase image of the same biologicalsample can be included in the sets 1402 a, 1402 b, and 1402 c,respectively.

In some embodiments, the plurality of training images are acquired toenable training of the GAN model such that the synthetic images (e.g.,synthetic fluorescence images, synthetic phase images) generated by theGAN model will provide the same performance in downstream analyses asreal images (e.g. real fluorescence images, real phase images). In someembodiments, the downstream analyses include a classification task thatclassifies an image as corresponding to one class out of M classes. Forexample, the classification task can involve classifying an image ascorresponding to a particular cell state out of multiple cell stateclasses (e.g., healthy state, diseased state). As another example, theclassification task can involve classifying an image as corresponding toa particular perturbation out of multiple perturbation classes. In orderto train the GAN model to generate synthetic images that can beclassified as accurately as the real images, the training images includeimages corresponding to the M classes (or conditions). For example, ifthe M classes include a healthy cell state class and a diseased cellstate class, the plurality of bright-field training images 1402 a caninclude bright-field images depicting healthy cells and bright-fieldimages depicting diseased cells, the plurality of fluorescence trainingimages 1402 b can include fluorescence images depicting healthy cellsand fluorescence images depicting diseased cells, and the plurality ofphase training images 1402 c can include phase images depicting healthycells and phase images depicting diseased cells. For example, if the Mclasses include M perturbations, the plurality of bright-field trainingimages 1402 a can include bright-field images depicting the Mperturbations, etc. Each training image can be labelled with thecorresponding condition. For example, a phase image depicting a diseasedcell state can be associated with a diseased label.

In an exemplary implementation, the plurality of training imagesincludes X fields of view per condition. i.e., M×X fields of view intotal. Specifically, in the plurality of bright-field images 1402 a,each field of view includes N bright-field images captured using Nillumination settings, thus resulting in M×X×N bright-field images intotal. In the plurality of fluorescence images 1402 b, each field ofview includes one fluoresce image, thus resulting in M×X fluorescenceimages in total. In the plurality of phase images 1402 c, each field ofview includes one phase image, thus resulting in M×X phase images intotal. In some embodiments, the bright-field images are atmagnification=m_(in), while the fluorescence images are at magnificationm_(out)>=m_(in) and the phase images are at m_(out)>=m_(in).

At block 1404, the system trains a classifier configured to receive aninput image and output a classification result indicative of one of Mconditions. For example, if the M conditions include a healthy conditionand a diseased condition, the classifier is configured to receive aninput image and output a classification result indicative of either thehealthy condition or the diseased condition. After the classifier istrained, it is used during the training of the GAN model to ensure thatthe GAN model can generate synthetic image data that can be classifiedto the same or similar level of accuracy as real image data, asdescribed below.

In some embodiments, the classifier is trained using the same type ofimage data that the GAN model is configured to output. In the depictedexample in FIGS. 14A and 14B, the GAN model is configured to output afluorescence image and a phase image; thus, the classifier trained inblock 1404 is trained using the plurality of fluorescence trainingimages 1402 b and the plurality of phase training images 1402 c. Duringtraining, each fluorescence image or phase image is inputted into theclassifier to obtain a predicted classification result (e.g., healthy ordiseased). The predicted classification result is then compared againstthe actual class associated with the training image (e.g., whethertraining image in fact depicts a healthy cell state or diseased cellstate) and, based on the comparison, the classifier can be updatedaccordingly. The classifier can be implemented using any classificationalgorithm, such as a logistic regression model, a naïve Bayes model, adecision tree model, a random forest model, a support vector machinemodel, etc.

At block 1406, the system trains the GAN model based on the trainingimages. Block 1406 can include steps 1408 a-1408 e, which can berepeated until the training is complete (e.g., when convergence isreached). The steps 1408 a-e are described below with reference to FIG.15, which is a schematic diagram illustrating the steps, in accordancewith some embodiments.

As shown in FIG. 15, the GAN model includes a multi-head attention layer1502 comprising a matrix of weights, specifically, K sets of weightsw₁-w_(n), a generator 1504, a discriminator 1508, and the trainedclassifier 1506 from block 1404. During training of the GAN model, theclassifier remains fixed while the generator, the discriminator, and theattention layer are updated, as described below.

At block 1408 a, the system applies each of K sets of weights in theattention layer of the GAN model to a set of bright-field trainingimages. The set of bright-field training images are obtained from theplurality of bright-field images 1402 a. In some embodiments, the set ofbright-field training images correspond to the same field of view anddepict the same biological sample, but are captured using differentillumination settings. For example, if a LED array comprises Nillumination emitters (e.g., LEDs), each illumination emitter can beturned on one at a time and a bright-field image of the biologicalsample illuminated by each illumination emitter can be captured, thusresulting in a set of N bright-field training images.

In the depicted example in FIG. 15, the system receives a set of Nbright-field training images corresponding to illumination settings 1-N.For example, the first bright-field image depicts the biological samplebeing illuminated using the illumination setting 1 (e.g., only the firstLED in the array is turned on), the second bright-field image depictsthe biological sample being illuminated using the illumination setting 2(e.g., only the second LED in the array is turned on), . . . and theN-th bright-field image depicts the biological sample being illuminatedusing the illumination setting N (e.g., only the N-th LED in the arrayis turned on).

The attention layer generates K sets of weights and each set comprises Nweights. Each set of weights w₁-w_(n) is applied to the N images togenerate an aggregated image. For each set of weights, the attentionlayer 1502 assigns a continuous weight (e.g., a normalized scalarweight) in the set to each of the set of bright-field training images.These weights correspond to intensity values of the correspondingillumination settings (e.g., the corresponding LEDs). As shown, w₁ isapplied to (e.g., multiplied with) the first bright-field image, w₂ isapplied to the second bright-field image, w_(n) is applied to the N-thbright-field image. After the weights are applied, the weighted imagescan be aggregated (e.g., summed) to obtain one aggregated bright-fieldimage. Because there are K sets of weights, K aggregated images 1512 canbe generated. In some embodiments, the attention layer is an adaptedmulti-head attention layer. The attention mechanism allows the naturalgeneration of K linear combination of bright-field images (i.e.,aggregated images). These aggregated images are fed to the rest of thenetwork as described herein.

At block 1408 b, the system inputs each of the aggregated bright-fieldimages into the GAN model. With reference to FIG. 15, each of theaggregated bright-field images 1512 is inputted into the generator 1504,which outputs a synthetic fluorescence image 1514 a and a syntheticphase image 1514 b. The generator can be implemented in a similar manneras the generator described above with reference to FIGS. 3A-D.

During training, the generator output (i.e., the generated fluorescenceimage 1514 a and the generated phase image 1514 b) can be connecteddirectly to the discriminator input. The discriminator 1508 is trainedto distinguish generated images and real images. During training, agenerated image can be inputted into the discriminator 1508 to obtain adiscriminator loss and a generator loss as described above. Further, areal image can also be inputted into the discriminator 1508 to generatea discriminator loss and a generator loss. The real image can be thereal fluorescence image 1516 a (from the plurality of fluorescencetraining images 1402 b in FIG. 14A) corresponding to the same field ofview as the input bright-field images, or the real phase image 1516 b(from the plurality of phase training images 1402 c in FIG. 14A)corresponding to the same field of view as the input bright-fieldimages.

In some embodiments, the discriminator loss function is a Wassersteindiscriminator loss and calculated as follows:

${\nabla_{w}\frac{1}{m}}{\sum_{i = 1}^{m}\left\lbrack {{f\left( x^{(i)} \right)} - {f\left( {G\left( z^{(i)} \right)} \right)}} \right.}$

where f(x) is the discriminator's output based on wavelet coefficientsof a real fluorescence or phase image, w is the model weights of thediscriminator, m is the size of the mini-batch, f is the discriminatormodel, x is the real image, z is the input (bright-field image 1512), Gis the generator model, and f(G(z)) is the discriminator's output basedon the predicted wavelet coefficients corresponding to a syntheticfluorescence or phase image.

In some embodiments, the generator loss function is a Wassersteingenerator loss and calculated as follows:

${\nabla_{\theta}\frac{1}{m}}{\sum_{i = 1}^{m}\left\lbrack {f\left( {G\left( z^{(i)} \right)} \right)} \right.}$

where f(x) is the discriminator's output based on wavelet coefficientsof a real fluorescence or phase image, m is the size of the mini-batch,f is the discriminator model, z is the input (bright-field image 1512),G is the generator model, and f(G(z)) is the discriminator's outputbased on the predicted wavelet coefficients.

At block 1408 c, the system inputs each generated image and a real imagecorresponding to the generated image into the trained classifier toobtain a classifier loss. For example, the generated fluorescence image1514 a is inputted into the classifier 1506 to obtain a firstclassification result; the real fluorescence image 1516 a is inputtedinto the classifier 1506 to obtain a second classification result; aclassifier loss can be calculated based on the difference between thefirst classification result and the second classification result. Asanother example, the generated phase image 1514 b is inputted into theclassifier 1506 to obtain a third classification result; the real phaseimage 1516 b is inputted into the classifier 1506 to obtain a fourthclassification result; a classifier loss can be calculated based on thedifference between the third classification result and the fourthclassification result.

At block 1408 d, the system augments the generator loss based on theclassifier loss. For example, the generator loss can be augmented with aL2 norm of the classification score from the real images. In someembodiments, if no classifier is available or no classification iswanted, the classifier loss is replaced by a constant (e.g. 0).

At block 1408 e, the system updates the GAN model based on the augmentedgenerator loss. The backpropagation follows the same procedure asdescribed above. The discriminator updates its weights throughback-propagation based on the discriminator loss through thediscriminator network. Further, the augmented generator loss isback-propagated to update the weights in the attention layer (e.g., theweights with which the aggregated image is calculated) and thegenerator. For example, the generated loss calculated based on anaggregated image corresponding to the K-th set of weights can be used toupdate the K-th set of weights in the attention layer.

At block 1410, the system obtains one or more optimal illuminationpatterns based on the weights in the attention layer of the trained GANmodel. As described above, the weights in the attention layer (e.g.,w₁-w_(n)) can be indicative of the intensity values of the correspondingillumination settings (e.g., the corresponding LEDs in the LED array).As discussed above, the attention layer can be a multi-head attentionlayer that provide K sets of weights (i.e., K linear combinations), thusresulting in K illumination patterns.

In some embodiments, before the block 1406, the generator and thediscriminator of the GAN model are pre-trained using bright-field imagesin which all LEDs in the LED array are turned on. After thepre-training, block 1406 is performed to update the attention weightswhile the generator and the discriminator remain fixed. The optimalcombination of the illuminations can be obtained based on the updatedweights.

FIG. 16A illustrates synthetic images generated by an exemplary GANmodel to study the NASH disease, in accordance with some embodiments. Inthe depicted example, Hepg2 cells with NASH genetic background areilluminated using an optimal illumination pattern identified usingtechniques described herein, and a bright-field image is captured. Thebright-field image is inputted into a GAN model, which outputs agenerated phase image and a generated fluorescence/bodipy image in FIG.16A. The GAN model used in FIG. 16A can be configured to include atrained classifier as described with reference to FIGS. 14A-B and 15.The classifier has two conditions: a healthy condition (e.g., noperturbations) and a diseased condition (e.g., inflammation cocktailplus fatty acid). As described, the GAN model can be trained such thatthe generated images can be classified by the classifier to the same orsimilar degree of accuracy as real images.

The synthetic images generated by the GAN model can be used to createdisease models. FIG. 16B illustrates downstream analyses of thegenerated images, in accordance with some embodiments. As shown, thegenerated images can be used to perform nucleus segmentation and blobdetection (e.g., using image-processing algorithms). Thus, the generatedimages can be used to create a disease model for the NASH disease andstudy chemical perturbations in NASH.

The synthetic images generated by the GAN model can also be used toevaluate the efficacy of a treatment. The system processes three groupsof generated images: a first group of images depicting healthy tissuesthat do not have the disease, a second group of images depictinguntreated diseased tissues, and a third group of images depictingdiseased tissues that have been treated (e.g., using a particular drug).In the depicted example in FIG. 16C, the first group of images (labelled“untreated”) comprises images of healthy tissues that do not have thenon-alcoholic steatohepatitis (NASH) disease, the second group of images(labelled “NASH 2X”) comprises images of untreated tissues having theNASH disease, and the third group of images (labelled “NASH 2X+ACCinhibitor”) comprise images of tissues with NASH that have been treatedwith a drug (e.g., 50 uM ACC inhibitor or firsocostate). In someembodiments, the three groups of images capture tissues of the samesubject at different times. In some embodiments, the three groups ofimages capture tissues of different subjects. The images can be phaseimages or fluorescence images generated using the techniques describedherein (e.g., from bright-field images). Each phase or fluorescenceimage can be generated based on multiple bright-field images and thuscomprises richer information such as phase shift information.Accordingly, the generated phase or fluorescence images can allow forhigher precision in downstream analyses, as described below.

Specifically, to evaluate the efficacy of the drug, the system generatesa distribution for each group of images to determine whether thedistributions reflect an effect of the drug on the disease state. InFIG. 16C, three probability distributions are generated using aclassifier trained on generated images. The X-axis indicates theprobability outputted by the classifier upon receiving an input image(e.g., a generated phase image). As shown by the three distributions,generated images of healthy tissues (i.e., distribution 1620) aregenerally classified as having lower probabilities of having the diseasethan generated images of the diseased tissues (i.e., distribution 1624).Further, generated images of treated tissues (i.e., distribution 1622)are generally classified as having lower probabilities of having thedisease than generated images of diseased tissues. A comparison of thesedistributions may indicate that the drug is effective at reversing orreducing the disease state, because the treatment has caused thediseased tissues to include features more similar to the healthy stateand less similar to the diseased state. Thus, the synthetic images canenable biophysics understanding of the distribution of lipids in thecells and provide insight for treatment.

In some embodiments, rather than using distributions, the system canidentify image clusters in an embedding space (e.g., UMAP), as shown inFIG. 16D. In the UMAP, each point represents an image embedding of animage (e.g., a generated phase image). As shown, the generated images oftreated tissues form a cluster that moves away from the cluster ofgenerated diseased images and toward the cluster of generated healthyimages, which may indicate that the drug is effective at reversing orreducing the disease state.

The analysis in FIGS. 16C and 16D can applied to evaluate a plurality ofdrug candidates with respect to a disease of interest. For example,tissues treated by each drug candidate can be imaged, for example, by amicroscope, to obtain bright-field images. The bright-field images canbe transformed into phase images using the techniques described herein.Distribution or cluster can be generated for phase images of eachtreatment. Accordingly, the resulting plot can comprise a distributionor cluster representing the disease state, a distribution or clusterrepresenting the health state, and a plurality of distributions orclusters each representing a candidate drug. The system can thenidentify the distribution or cluster closest to the healthy state toidentify the most effective candidate drug candidate.

FIG. 17A illustrates synthetic images generated by an exemplary GANmodel to study tuberous sclerosis (“TSC”), in accordance with someembodiments. In the depicted example, NGN2 neurons are illuminated usingan optimal illumination pattern identified using techniques describedherein, and bright-field images are captured. The bright-field imagesare inputted into a GAN model, which outputs generated phase images andgenerated fluorescence/bodipy images in FIG. 17A. The GAN model used inFIG. 17A can be configured to include a trained classifier as describedwith reference to FIGS. 14A-B and 15. The classifier has 2 conditions: ahealthy condition (e.g., wild type) and a diseased condition (e.g., TSCKO). As described, the GAN model can be trained such that the generatedimages can be classified by the classifier to the same or similar degreeof accuracy as real images.

The synthetic images generated by the GAN model can also be used toevaluate the efficacy of a treatment. The system processes three groupsof generated images: a first group of images depicting healthy tissuesthat do not have the disease, a second group of images depictinguntreated diseased tissues, and a third group of images depictingdiseased tissues that have been treated (e.g., using a particular drug).In the depicted example in FIG. 17B, the first group of images (labelled“Wildtype”) comprises images of healthy tissues that do not have the TSCdisease, the second group of images (labelled “TSC”) comprises images ofuntreated tissues having the TSC disease, and the third group of images(labelled “TSC+Rapamycin”) comprise images of tissues with TSC that havebeen treated with a drug. In some embodiments, the three groups ofimages capture tissues of the same subject at different times. In someembodiments, the three groups of images capture tissues of differentsubjects. The images are phase images generated using the techniquesdescribed herein (e.g., from bright-field images).

Specifically, to evaluate the efficacy of the drug, the system generatesa distribution for each group of images to determine whether thedistributions reflect an effect of the drug on the disease. In FIG. 17B,three biomarker distributions are generated. As shown by thesedistributions, the drug appears to be effective at reversing or reducingthe disease state, because the treatment has caused the diseased tissuesto include features more similar to the healthy state and less similarto the diseased state. The analysis in FIG. 17B can applied to evaluatea plurality of drug candidates with respect to a disease of interest, asdescribed above.

Exemplary methods, non-transitory computer-readable storage media,systems, and electronic devices are set out in the following items:

-   -   1. A method for training a machine-learning model to generate        images of biological samples, comprising:        -   obtaining a plurality of training images comprising:            -   a training image of a first type, and            -   a training image of a second type;        -   generating, based on the training image of the first type, a            plurality of wavelet coefficients using the machine-learning            model;        -   generating, based on the plurality of wavelet coefficients,            a synthetic image of the second type;        -   comparing the synthetic image of the second type with the            training image of the second type; and        -   updating the machine-learning model based on the comparison.    -   2. The method of item 1, wherein the training image of the first        type is a bright-field image of a biological sample.    -   3. The method of item 2, wherein the training image of the        second type is a fluorescence image of the biological sample.    -   4. The method of any of items 1-3, wherein the machine-learning        model comprises a generator and a discriminator.    -   5. The method of item 4, wherein the machine-learning model        comprises a conditional GAN model.    -   6. The method of any of items 4-5, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   7. The method of item 6, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   8. The method of any of items 6-7, wherein the plurality of        neural networks comprises a plurality of U-Net neural networks.    -   9. The method of any of items 5-8, wherein the discriminator is        a PatchGAN neural network.    -   10. The method of any of items 1-9, further comprising:        generating, based on the training image of the first type, an        image of a third type.    -   11. The method of item 10, wherein the image of the third type        is a phase shift image.    -   12. The method of any of items 1-11, further comprising:        generating, based on the training image of the first type, an        image of a fourth type.    -   13. The method of item 12, wherein the image of the fourth type        comprises segmentation data.    -   14. The method of any of items 1-13, wherein the training image        of the first type is captured using a microscope according to a        first illumination scheme.    -   15. The method of item 14, wherein the first illumination scheme        comprises one or more illumination patterns.    -   16. The method of any of items 14-15, wherein the training image        of the first type is part of a bright-field image array.    -   17. The method of any of items 14-16, wherein the plurality of        training images is a first plurality of training images, the        method further comprising:        -   based on the comparison, identifying a second illumination            scheme;        -   obtaining a second plurality of training images comprising            one or more images of the first type, wherein the one or            more images of the first type are obtained based on the            second illumination scheme;        -   training the machine-learning model based on the second            plurality of training images.    -   18. The method of any of items 1-16, further comprising:        -   obtaining, using a microscope, a plurality of images of the            first type; and        -   generating, based on the obtained plurality of images, a            plurality of synthetic images of the second type using the            machine-learning model.    -   19. The method of item 18, further comprising: training a        classifier based on the plurality of synthetic images of the        second type.    -   20. The method of item 19, wherein the microscope is a first        microscope, wherein the classifier is a first classifier,        further comprising:        -   obtaining, using a second microscope, a plurality of images            of the second type;        -   training a second classifier based on the plurality of            images of the second type;        -   comparing performance of the first classifier and the second            classifier.    -   21. The method of item 20, wherein the second microscope is a        fluorescence microscope.    -   22. A method for generating enhanced images of biological        samples, comprising:        -   obtaining, using a microscope, an image of a biological            sample; and        -   generating, based on the image, an enhanced image of the            biological sample using a machine-learning model, wherein            the machine-learning model has been trained by:            -   obtaining a plurality of training images comprising                -   a training image of a first type, and                -   a training image of a second type;            -   generating, based on the training image of the first                type, a plurality of wavelet coefficients using the                machine-learning model;            -   generating, based on the plurality of wavelet                coefficients, a synthetic image of the second type;            -   comparing the synthetic image of the second type with                the training image of the second type; and            -   updating the machine-learning model based on the                comparison.    -   23. The method of item 22, wherein the training image of the        first type is a bright-field image of a biological sample.    -   24. The method of item 22, wherein the training image of the        second type is a fluorescence image of the biological sample.    -   25. The method of any of items 22-24, wherein the        machine-learning model comprises a generator and a        discriminator.    -   26. The method of item 25, wherein the machine-learning model        comprises a conditional GAN model.    -   27. The method of any of items 25-26, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   28. The method of item 27, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   29. The method of any of items 27-28, wherein the plurality of        neural networks comprises a plurality of U-Net neural networks.    -   30. The method of any of items 26-29, wherein the discriminator        is a PatchGAN neural network.    -   31. The method of any of items 23-30, further comprising:        generating, based on the training image of the first type, an        image of a third type.    -   32. The method of item 31, wherein the image of the third type        is a phase shift image.    -   33. The method of any of items 23-32, further comprising:        generating, based on the training image of the first type, an        image of a fourth type.    -   34. The method of item 33, wherein the image of the fourth type        comprises segmentation data.    -   35. The method of any of items 23-34, wherein the training image        of the first type is captured using a microscope according to a        first illumination scheme.    -   36. The method of item 35, wherein the first illumination scheme        comprises one or more illumination patterns.    -   37. The method of any of items 35-36, wherein the training image        of the first type is part of a bright-field image array.    -   38. The method of any of items 35-37, wherein the plurality of        training images is a first plurality of training images, the        method further comprising:        -   based on the comparison, identifying a second illumination            scheme;        -   obtaining a second plurality of training images comprising            one or more images of the first type, wherein the one or            more images of the first type are obtained based on the            second illumination scheme;        -   training the machine-learning model based on the second            plurality of training images.    -   39. The method of any of items 35-38, further comprising:        -   obtaining, using a microscope, a plurality of images of the            first type; and        -   generating, based on the obtained plurality of images, a            plurality of synthetic images of the second type using the            machine-learning model.    -   40. The method of item 39, further comprising: training a        classifier based on the plurality of synthetic images of the        second type.    -   41. The method of item 40, wherein the microscope is a first        microscope, wherein the classifier is a first classifier,        further comprising:        -   obtaining, using a second microscope, a plurality of images            of the second type;        -   training a second classifier based on the plurality of            images of the second type;        -   comparing performance of the first classifier and the second            classifier.    -   42. The method of item 41, wherein the second microscope is a        fluorescence microscope.    -   43. A system for training a machine-learning model to generate        images of biological samples, comprising:        -   a computing system comprising one or more processors, and            one or more memories storing a machine-learning model,            wherein the computing system is configured to receive a            plurality of training images of a first type and one a            training image of a second type, and wherein the computing            system is configured to:            -   generate, based on the training images of the first                type, a plurality of wavelet coefficients using the                machine-learning model;            -   generate, based on the plurality of wavelet                coefficients, a synthetic image of the second type;            -   compare the synthetic image of the second type with the                training image of the second type; and            -   update the machine-learning model based on the                comparison.    -   44. The system of item 43, wherein the training image of the        first type is a bright-field image of a biological sample.    -   45. The system of any of items 43-44, wherein the training image        of the second type is a fluorescence image of the biological        sample.    -   46. The system of any of items 43-45, wherein the        machine-learning model comprises a generator and a        discriminator.    -   47. The system of item 46, wherein the machine-learning model        comprises a conditional GAN model.    -   48. The system of any of items 46-47, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   49. The system of item 48, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   50. The system of any of items 48-49, wherein the plurality of        neural networks comprises a plurality of U-Net neural networks.    -   51. The system of any of items 46-50, wherein the discriminator        is a PatchGAN neural network.    -   52. The system of any of items 43-51, wherein the computing        system is further configured to: generate, based on the training        image of the first type, an image of a third type.    -   53. The system of item 52, wherein the image of the third type        is a phase shift image.    -   54. The system of any of items 43-53, wherein the computing        system is further configured to: generate, based on the training        image of the first type, an image of a fourth type.    -   55. The system of item 54, wherein the image of the fourth type        comprises segmentation data.    -   56. The system of any of items 43-55, wherein the training image        of the first type is captured using a microscope according to a        first illumination scheme.    -   57. The system of item 56, wherein the first illumination scheme        comprises one or more illumination patterns.    -   58. The system of any of items 56-57, wherein the training image        of the first type is part of a bright-field image array.    -   59. The system of any of items 56-58, wherein the plurality of        training images is a first plurality of training images, and        wherein the computing system is further configured to:        -   based on the comparison, identify a second illumination            scheme;        -   obtain a second plurality of training images comprising one            or more images of the first type, wherein the one or more            images of the first type are obtained based on the second            illumination scheme;        -   train the machine-learning model based on the second            plurality of training images.    -   60. The system of any of items 43-59, wherein the computing        system is further configured to:        -   obtain, using a microscope, a plurality of images of the            first type; and        -   generate, based on the obtained plurality of images, a            plurality of synthetic images of the second type using the            machine-learning model.    -   61. The system of item 59, wherein the computing system is        further configured to: train a classifier based on the plurality        of synthetic images of the second type.    -   62. The system of item 61, wherein the microscope is a first        microscope, wherein the classifier is a first classifier,        wherein the computing system is further configured to:        -   obtain, using a second microscope, a plurality of images of            the second type;        -   train a second classifier based on the plurality of images            of the second type;        -   compare performance of the first classifier and the second            classifier.    -   63. The system of item 62, wherein the second microscope is a        fluorescence microscope.    -   64. A system for generating enhanced images of biological        samples, comprising:        -   a computing system comprising one or more processors, and            one or more memories storing a machine-learning model,            wherein the computing system is configured to receive an            image of a biological sample obtained from a microscope and            generate, based on the image, an enhanced image of the            biological sample using a machine-learning model, wherein            the machine-learning model has been trained by:            -   obtaining a plurality of training images comprising:                -   a training image of a first type, and                -   a training image of a second type;            -   generating, based on the training image of the first                type, a plurality of wavelet coefficients using the                machine-learning model;            -   generating, based on the plurality of wavelet                coefficients, a synthetic image of the second type;            -   comparing the synthetic image of the second type with                the training image of the second type; and            -   updating the machine-learning model based on the                comparison.    -   65. The system of item 64, wherein the training image of the        first type is a bright-field image of a biological sample.    -   66. The system of item 64, wherein the training image of the        second type is a fluorescence image of the biological sample.    -   67. The system of any of items 64-66, wherein the        machine-learning model comprises a generator and a        discriminator.    -   68. The system of item 67, wherein the machine-learning model        comprises a conditional GAN model.    -   69. The system of any of items 67-68, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   70. The system of item 69, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   71. The system of any of items 69-70, wherein the plurality of        neural networks comprises a plurality of U-Net neural networks.    -   72. The system of any of items 67-71, wherein the discriminator        is a PatchGAN neural network.    -   73. The system of any of items 64-72, wherein the machine        learning model is further trained by generating, based on the        training image of the first type, an image of a third type.    -   74. The system of item 73, wherein the image of the third type        is a phase shift image.    -   75. The system of any of items 64-74, wherein the        machine-learning model has been trained by: generating, based on        the training image of the first type, an image of a fourth type.    -   76. The system of item 75, wherein the image of the fourth type        comprises segmentation data.    -   77. The system of any of items 64-76, wherein the training image        of the first type is captured using a microscope according to a        first illumination scheme.    -   78. The system of item 77, wherein the first illumination scheme        comprises one or more illumination patterns.    -   79. The system of any of items 77-78, wherein the training image        of the first type is part of a bright-field image array.    -   80. The system of any of items 77-79, wherein the plurality of        training images is a first plurality of training images, wherein        the machine-learning model has been trained by:        -   based on the comparison, identifying a second illumination            scheme;        -   obtaining a second plurality of training images comprising            one or more images of the first type, wherein the one or            more images of the first type are obtained based on the            second illumination scheme;        -   training the machine-learning model based on the second            plurality of training images.    -   81. The system of any of items 77-80, wherein the        machine-learning model has been trained by:        -   obtaining, using a microscope, a plurality of images of the            first type; and        -   generating, based on the obtained plurality of images, a            plurality of synthetic images of the second type using the            machine-learning model.    -   82. The system of item 81, wherein the machine-learning model        has been trained by: training a classifier based on the        plurality of synthetic images of the second type.    -   83. The system of item 82, wherein the microscope is a first        microscope, wherein the classifier is a first classifier,        wherein the machine-learning model has been trained by:        -   obtaining, using a second microscope, a plurality of images            of the second type;        -   training a second classifier based on the plurality of            images of the second type;        -   comparing performance of the first classifier and the second            classifier.    -   84. The system of item 83, wherein the second microscope is a        fluorescence microscope.    -   85. A method of processing images of a biological sample to        obtain one or more output images, comprising:        -   obtaining a plurality of images of the biological sample            using a plurality of configurations of a SLM of an optical            system, wherein the SLM is located in an optical path            between the biological sample and an image recording device;            and        -   inputting the plurality of images of the biological sample            into a trained machine-learning model to obtain the one or            more outputs images.    -   86. The method of item 85, wherein at least one configuration of        the plurality of configurations of the SLM is to generate one or        more optical aberrations.    -   87. The method of item 86, wherein generating one or more        optical aberrations comprises a spherical aberration,        astigmatism, defocus, distortion, tilt, or any combination        thereof    -   88. The method of any of items 85-86, wherein at least one        configuration of the plurality of configurations of the SLM is        to enhance one or more features.    -   89. The method of item 88, wherein the one or more features        comprise a cell border, an actin filament, nuclear shape,        cytoplasm segmentation, or any combination thereof    -   90. The method of any of items 85-89, wherein at least one        configuration of the plurality of configurations of the SLM is        to reduce optical aberrations.    -   91. The method of any of items 85-90, wherein the plurality of        SLM configurations is to obtain images of the biological sample        at different depths.    -   92. The method of any of items 85-91, wherein the        machine-learning model is configured to generate, based on an        image of a first type, an image of a second type.    -   93. The method of item 92, wherein the first type of images are        bright-field images.    -   94. The method of item 92, wherein the second type of images are        fluorescence images.    -   95. The method of item 92, wherein the second type of images are        enhanced versions of the first type of images.    -   96. The method of any of items 92-95, wherein the        machine-learning model is a GAN model or a self-supervised        model.    -   97. The method of any of items 85-96, wherein the plurality of        images are obtained using a plurality of configurations of a        light source of the optical system.    -   98. The method of item 97, wherein the light source is a LED        array of the optical system.    -   99. The method of any of items 85-98, wherein at least one        configuration of the plurality of SLM configurations is obtained        by:        -   training the machine-learning model;        -   evaluating the trained machine-learning model; and        -   identifying the at least one configuration based on the            evaluation.    -   100. The method of any of items 85-99, wherein the trained        machine-learning model is configured to receive an input image        and output an enhanced version of the input image.    -   101. The method of item 100, wherein the enhanced version of the        input image comprises one or more enhanced cellular phenotypes.    -   102. An electronic device for processing images of a biological        sample to obtain one or more output images, comprising:        -   one or more processors;        -   a memory; and        -   one or more programs, wherein the one or more programs are            stored in the memory and configured to be executed by the            one or more processors, the one or more programs including            instructions for:            -   obtaining a plurality of images of the biological sample                using a plurality of configurations of a SLM of an                optical system, wherein the SLM is located in an optical                path between the biological sample and an image                recording device; and            -   inputting the plurality of images of the biological                sample into a trained machine-learning model to obtain                the one or more output images.    -   103. A non-transitory computer-readable storage medium storing        one or more programs for processing images of a biological        sample to obtain one or more output images, the one or more        programs comprising instructions, which when executed by one or        more processors of an electronic device, cause the electronic        device to:        -   obtain a plurality of images of the biological sample using            a plurality of configurations of a SLM of an optical system,            wherein the SLM is located in an optical path between the            biological sample and an image recording device; and input            the plurality of images of the biological sample into a            trained machine-learning model to obtain the one or more            output images.    -   104. A method of classifying images of a biological sample,        comprising:        -   obtaining a plurality of images of the biological sample            using a plurality of configurations of an SLM of an optical            system, wherein the SLM is located in an optical path            between the biological sample and an image recording device;            and        -   inputting the plurality of images of the biological sample            into a trained machine-learning model to obtain one or more            classification outputs.    -   105. The method of item 104, wherein at least one configuration        of the plurality of configurations of the SLM is to generate one        or more optical aberrations.    -   106. The method of item 105, wherein generating one or more        optical aberrations comprises a spherical aberration,        astigmatism, defocus, distortion, tilt, or any combination        thereof    -   107. The method of any of items 104-106, wherein at least one        configuration of the plurality of configurations of the SLM is        to enhance one or more features.    -   108. The method of item 107, wherein the one or more features        comprise a cell border, an actin filament, nuclear shape,        cytoplasm segmentation, or any combination thereof    -   109. The method of any of items 104-108, wherein at least one        configuration of the plurality of configurations of the SLM is        to reduce optical aberrations.    -   110. The method of any of items 104-109, wherein the plurality        of SLM configurations is to obtain images of the biological        sample at different depths.    -   111. The method of any of items 104-110, wherein the plurality        of images are obtained using a plurality of configurations of a        light source of the optical system.    -   112. The method of item 111, wherein the light source is a LED        array of the optical system.    -   113. The method of any of items 104-112, wherein at least one        configuration of the plurality of SLM configurations is obtained        by:        -   training the machine-learning model;        -   evaluating the trained machine-learning model; and        -   identifying the at least one configuration based on the            evaluation.    -   114. The method of any of items 104-113, wherein the trained        machine-learning model is configured to receive an input image        and detect one or more pre-defined objects in the input image.    -   115. The method of item 114, wherein the pre-defined objects        include a diseased tissue.    -   116. An electronic device for classifying images of a biological        sample, comprising:        -   one or more processors;        -   a memory; and        -   one or more programs, wherein the one or more programs are            stored in the memory and configured to be executed by the            one or more processors, the one or more programs including            instructions for:            -   obtaining a plurality of images of the biological sample                using a plurality of configurations of an SLM of an                optical system, wherein the SLM is located in an optical                path between the biological sample and an image                recording device; and            -   inputting the plurality of images of the biological                sample into a trained machine-learning model to obtain                one or more classification outputs.    -   117. A non-transitory computer-readable storage medium storing        one or more programs for classifying images of a biological        sample, the one or more programs comprising instructions, which        when executed by one or more processors of an electronic device,        cause the electronic device to:        -   obtain a plurality of images of the biological sample using            a plurality of configurations of an SLM of an optical            system, wherein the SLM is located in an optical path            between the biological sample and an image recording device;            and        -   input the plurality of images of the biological sample into            a trained machine-learning model to obtain one or more            classification outputs.    -   118. A method for training a machine-learning model, comprising:        -   obtaining a plurality of images of a biological sample using            a plurality of configurations of an SLM of an optical            system, wherein the SLM is located in an optical path            between the biological sample and an image recording device;            and        -   training the machine-learning model using the plurality of            images.    -   119. The method of item 118, wherein at least one configuration        of the plurality of configurations of the SLM is to generate one        or more optical aberrations.    -   120. The method of item 119, wherein generating one or more        optical aberrations comprises a spherical aberration,        astigmatism, defocus, distortion, tilt, or any combination        thereof    -   121. The method of any of items 118-120, wherein at least one        configuration of the plurality of configurations of the SLM is        to enhance one or more features.    -   122. The method of item 121, wherein the one or more features        comprise a cell border, an actin filament, nuclear shape,        cytoplasm segmentation, or any combination thereof    -   123. The method of any of items 118-122, wherein at least one        configuration of the plurality of configurations of the SLM is        to reduce optical aberrations.    -   124. The method of any of items 118-123, wherein at least one        configuration of the plurality of configurations of the SLM is        to obtain images of the biological sample at different depths.    -   125. The method of any of items 118-124, wherein the        machine-learning model is configured to generate, based on an        image of a first type, an image of a second type.    -   126. The method of item 125, wherein the first type of images        are bright-field images.    -   127. The method of item 125, wherein the second type of images        are fluorescence images.    -   128. The method of any of items 118-127, wherein the        machine-learning model is a GAN model or a self-supervised        model.    -   129. The method of any of items 118-128, wherein the        machine-learning model is a classification model.    -   130. The method of any of items 118-129, wherein the plurality        of images are obtained using a plurality of configurations of a        light source of the optical system.    -   131. The method of item 130, wherein the light source is a LED        array of the optical system.    -   132. The method of any of items 118-131, wherein training the        machine-learning model comprises:        -   (a) training the machine-learning model using a first image,            wherein the first image is obtained using a first            configuration of the SLM of the optical system;        -   (b) evaluating the trained machine-learning model;        -   (c) based on the evaluation, identifying a second            configuration of the SLM; and        -   (d) training the machine-learning model using a second            image, wherein the second image is obtained using the second            configuration of the SLM of the optical system.    -   133. The method of item 112, wherein the evaluation is based on        a loss function of the machine-learning model.    -   134. The method of any of items 112-113, further comprising:        repeating steps (a)-(d) until a threshold is met.    -   135. The method of item 114, wherein the threshold is indicative        of convergence of the training.    -   136. The method of any of items 118-135, wherein the trained        machine-learning model is configured to receive an input image        and output an enhanced version of the input image.    -   137. The method of item 136, wherein the enhanced version of the        input image comprises one or more enhanced cellular phenotypes.    -   138. The method of any of items 118-135, wherein the trained        machine-learning model is configured to receive an input image        and detect one or more pre-defined objects in the input image.    -   139. The method of item 138, wherein the pre-defined objects        include a diseased tissue.    -   140. An electronic device for training a machine-learning model,        comprising:        -   one or more processors;        -   a memory; and        -   one or more programs, wherein the one or more programs are            stored in the memory and configured to be executed by the            one or more processors, the one or more programs including            instructions for:            -   obtaining a plurality of images of a biological sample                using a plurality of configurations of an SLM of an                optical system, wherein the SLM is located in an optical                path between the biological sample and an image                recording device; and            -   training the machine-learning model using the plurality                of images.    -   141. A non-transitory computer-readable storage medium storing        one or more programs for training a machine-learning model, the        one or more programs comprising instructions, which when        executed by one or more processors of an electronic device,        cause the electronic device to:        -   obtain a plurality of images of a biological sample using a            plurality of configurations of an SLM of an optical system,            wherein the SLM is located in an optical path between the            biological sample and an image recording device; and train            the machine-learning model using the plurality of images.    -   142. A method of generating enhanced images of biological        samples, comprising:        -   obtaining, using a microscope, an image of a biological            sample illuminated using an illumination pattern of an            illumination source, wherein the illumination pattern is            determined by:            -   training a classification model configured to receive an                input image and output a classification result,            -   training, using the trained classification model, a                machine-learning model having an plurality of weights                corresponding to a plurality of illumination settings,                and            -   identifying the illumination pattern based on the                plurality of weights of the trained machine-learning                model; and        -   generating an enhanced image of the biological sample by            inputting the obtained image of the biological sample into            the trained machine-learning model.    -   143. The method of item 142, wherein the obtained image is a        bright-field image.    -   144. The method of any of items 142-143, wherein the enhanced        image is a fluorescence image, a phase image, or a combination        thereof    -   145. The method of any of items 142-144, wherein the        illumination source comprises an array of illumination emitters.    -   146. The method of item 145, wherein the illumination source is        a LED array.    -   147. The method of any of items 142-146, wherein the        illumination pattern indicates whether each illumination emitter        is turned on or off and the intensity of each illumination        emitter.    -   148. The method of any of items 145-147, wherein each        illumination setting of the plurality of illumination settings        corresponds to a respective illumination emitter of the        illumination source; and wherein each weight corresponds to an        intensity of the respective illumination emitter.    -   149. The method of any of items 142-148, wherein the        classification model is configured to receive an input phase        image or an input fluorescence image and output a classification        result indicative of one class out of a plurality of pre-defined        classes.    -   150. The method of item 149, wherein the plurality of        pre-defined classes comprises a healthy class and a diseased        class.    -   151. The method of item 150, wherein the machine-learning model        is a GAN model comprising an attention layer comprising the        plurality of weights, a discriminator, and a generator.    -   152. The method of item 151, wherein the machine-learning model        is a conditional GAN model.    -   153. The method of any of items 151-152, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   154. The method of item 153, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   155. The method of any of items 153-154, wherein the plurality        of neural networks comprises a plurality of U-Net neural        networks.    -   156. The method of any of items 151-155, wherein the        discriminator is a PatchGAN neural network.    -   157. The method of any of items 151-156, wherein training, using        the trained classification model, the machine-learning model        comprises:        -   applying the plurality of weights to a plurality of            bright-field training images;        -   aggregating the plurality of weighted bright-field training            images into an aggregated bright-field image;        -   inputting the aggregated bright-field training image into            the machine-learning model to obtain an enhanced training            image and a generator loss;        -   inputting the enhanced training image into the trained            classifier to obtain a classifier loss;        -   augmenting the generator loss based on the classifier loss;            and        -   updating the plurality of weights based on the augmented            generator loss.    -   158. The method of any of items 142-157, further comprising:        classifying the enhanced image using the trained classifier.    -   159. The method of any of items 142-158, further comprising:        displaying the enhanced image.    -   160. A system for generating enhanced images of biological        samples, comprising:        -   one or more processors;        -   a memory; and        -   one or more programs, wherein the one or more programs are            stored in the memory and configured to be executed by the            one or more processors, the one or more programs including            instructions for:            -   obtaining, using a microscope, an image of a biological                sample illuminated using an illumination pattern of an                illumination source, wherein the illumination pattern is                determined by:                -   training a classification model configured to                    receive an input image and output a classification                    result,                -   training, using the trained classification model, a                    machine-learning model having an plurality of                    weights corresponding to a plurality of illumination                    settings, and                -   identifying the illumination pattern based on the                    plurality of weights of the trained machine-learning                    model; and            -   generating an enhanced image of the biological sample by                inputting the obtained image of the biological sample                into the trained machine-learning model.    -   161. A non-transitory computer-readable storage medium storing        one or more programs for generating enhanced images of        biological samples, the one or more programs comprising        instructions, which when executed by one or more processors of        an electronic device, cause the electronic device to:        -   obtain, using a microscope, an image of a biological sample            illuminated using an illumination pattern of an illumination            source, wherein the illumination pattern is determined by:            -   training a classification model configured to receive an                input image and output a classification result,            -   training, using the trained classification model, a                machine-learning model having an plurality of weights                corresponding to a plurality of illumination settings,                and            -   identifying the illumination pattern based on the                plurality of weights of the trained machine-learning                model; and        -   generate an enhanced image of the biological sample by            inputting the obtained image of the biological sample into            the trained machine-learning model.    -   162. A method of evaluating a treatment with respect to a        disease of interest, comprising:        -   receiving a first plurality of images depicting a first set            of healthy biological samples not affected by the disease of            interest;        -   receiving a second plurality of images depicting a second            set of untreated biological samples affected by the disease            of interest;        -   receiving a third plurality of images depicting a third set            of treated biological samples affected by the disease of            interest and treated by the treatment;        -   inputting the first plurality of images into a trained            machine-learning model to obtain a first plurality of            enhanced images;        -   inputting the second plurality of images into the trained            machine-learning model to obtain a second plurality of            enhanced images;        -   inputting the third plurality of images into the trained            machine-learning model to obtain a third plurality of            enhanced images;        -   comparing the first plurality of enhanced images, the second            plurality of enhanced images, and the third plurality of            enhanced images to evaluate the treatment.    -   163. The method of item 162, wherein the first plurality of        images, the second plurality of images, and the third plurality        of images are bright-field images.    -   164. The method of any of items 162-163, wherein the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images are        fluorescence images.    -   165. The method of any of items 162-163, wherein the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images are phase        images.    -   166. The method of any of items 162-165, wherein comparing the        first plurality of enhanced images, the second plurality of        enhanced images, and the third plurality of enhanced images to        evaluate the treatment comprises: identifying, in each image, a        signal associated with a biomarker.    -   167. The method of item 166, wherein comparing the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images to evaluate        the treatment further comprises:        -   determining a first distribution based on signals of the            biomarker in the first plurality of enhanced images;        -   determining a second distribution based on signals of the            biomarker in the second plurality of enhanced images; and        -   determining a third distribution based on signals of the            biomarker in the third plurality of enhanced images.    -   168. The method of item 167, wherein comparing the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images to evaluate        the treatment further comprises:        -   comparing the first distribution, the second distribution,            and the third distribution to evaluate the treatment.    -   169. The method of any of items 162-165, wherein comparing the        first plurality of enhanced images, the second plurality of        enhanced images, and the third plurality of enhanced images to        evaluate the treatment comprises: determining, for each image, a        score indicative of the statement of the disease of interest.    -   170. The method of item 169, wherein comparing the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images to evaluate        the treatment further comprises:        -   determining a first distribution based on scores of the            first plurality of enhanced images;        -   determining a second distribution based on scores of the            second plurality of enhanced images; and        -   determining a third distribution based on scores of the            third plurality of enhanced images.    -   171. The method of item 170, wherein comparing the first        plurality of enhanced images, the second plurality of enhanced        images, and the third plurality of enhanced images to evaluate        the treatment further comprises:        -   comparing the first distribution, the second distribution,            and the third distribution to evaluate the treatment.    -   172. The method of any of items 162-171, wherein the treatment        is a first treatment, the method further comprising:        -   receiving a fourth plurality of images depicting a fourth            set of treated biological samples affected by the disease of            interest and treated by a second treatment;        -   inputting the fourth plurality of images into the trained            machine-learning model to obtain a fourth plurality of            enhanced images;        -   comparing the first plurality of enhanced images, the second            plurality of enhanced images, the third plurality of            enhanced images, and the fourth plurality of enhanced images            to compare the first treatment and the second treatment.    -   173. The method of item 172, further comprising: selecting a        treatment out of the first treatment and the second treatment        based on the comparison.    -   174. The method of item 173, further comprising: administering        the selected treatment.    -   175. The method of item 173, further comprising: providing a        medical recommendation based on the selected treatment.    -   176. The method of any of items 162-175, wherein the trained        machine-learning model is is a GAN model comprising a        discriminator and a generator.    -   177. The method of item 176, wherein the machine-learning model        is a conditional GAN model.    -   178. The method of any of items 176-177, wherein the generator        comprises a plurality of neural networks corresponding to a        plurality of frequency groups.    -   179. The method of item 178, wherein each neural network of the        plurality of neural networks is configured to generate wavelet        coefficients for a respective frequency group.    -   180. The method of any of items 176-179, wherein the        discriminator is a PatchGAN neural network.    -   181. A system for evaluating a treatment with respect to a        disease of interest, comprising:        -   one or more processors;        -   a memory; and        -   one or more programs, wherein the one or more programs are            stored in the memory and configured to be executed by the            one or more processors, the one or more programs including            instructions for:        -   receiving a first plurality of images depicting a first set            of healthy biological samples not affected by the disease of            interest;            -   receiving a second plurality of images depicting a                second set of untreated biological samples affected by                the disease of interest;            -   receiving a third plurality of images depicting a third                set of treated biological samples affected by the                disease of interest and treated by the treatment;            -   inputting the first plurality of images into a trained                machine-learning model to obtain a first plurality of                enhanced images;            -   inputting the second plurality of images into the                trained machine-learning model to obtain a second                plurality of enhanced images;            -   inputting the third plurality of images into the trained                machine-learning model to obtain a third plurality of                enhanced images;            -   comparing the first plurality of enhanced images, the                second plurality of enhanced images, and the third                plurality of enhanced images to evaluate the treatment.    -   182. A non-transitory computer-readable storage medium storing        one or more programs for evaluating a treatment with respect to        a disease of interest, the one or more programs comprising        instructions, which when executed by one or more processors of        an electronic device, cause the electronic device to:        -   receiving a first plurality of images depicting a first set            of healthy biological samples not affected by the disease of            interest;        -   receiving a second plurality of images depicting a second            set of untreated biological samples affected by the disease            of interest;        -   receiving a third plurality of images depicting a third set            of treated biological samples affected by the disease of            interest and treated by the treatment;        -   inputting the first plurality of images into a trained            machine-learning model to obtain a first plurality of            enhanced images;        -   inputting the second plurality of images into the trained            machine-learning model to obtain a second plurality of            enhanced images;        -   inputting the third plurality of images into the trained            machine-learning model to obtain a third plurality of            enhanced images;        -   comparing the first plurality of enhanced images, the second            plurality of enhanced images, and the third plurality of            enhanced images to evaluate the treatment.

Although the disclosure and examples have been fully described withreference to the accompanying figures, it is to be noted that variouschanges and modifications will become apparent to those skilled in theart. Such changes and modifications are to be understood as beingincluded within the scope of the disclosure and examples as defined bythe claims.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the techniques and their practical applications. Othersskilled in the art are thereby enabled to best utilize the techniquesand various embodiments with various modifications as are suited to theparticular use contemplated.

1-21. (canceled)
 22. A method for generating enhanced images ofbiological samples, comprising: obtaining, using a microscope, an imageof a biological sample; and generating, based on the image, an enhancedimage of the biological sample using a machine-learning model, whereinthe machine-learning model has been trained by: obtaining a plurality oftraining images comprising a training image of a first type, and atraining image of a second type; generating, based on the training imageof the first type, a plurality of wavelet coefficients using themachine-learning model; generating, based on the plurality of waveletcoefficients, a synthetic image of the second type; comparing thesynthetic image of the second type with the training image of the secondtype; and updating the machine-learning model based on the comparison.23. The method of claim 22, wherein the training image of the first typeis a bright-field image of a biological sample.
 24. The method of claim22, wherein the training image of the second type is a fluorescenceimage of the biological sample.
 25. The method of claim 22, wherein themachine-learning model comprises a generator and a discriminator. 26.The method of claim 25, wherein the machine-learning model comprises aconditional GAN model.
 27. The method of claim 25, wherein the generatorcomprises a plurality of neural networks corresponding to a plurality offrequency groups.
 28. The method of claim 27, wherein each neuralnetwork of the plurality of neural networks is configured to generatewavelet coefficients for a respective frequency group.
 29. The method ofclaim 27, wherein the plurality of neural networks comprises a pluralityof U-Net neural networks.
 30. The method of claim 26, wherein thediscriminator is a PatchGAN neural network.
 31. The method of claim 23,further comprising: generating, based on the training image of the firsttype, an image of a third type.
 32. The method of claim 31, wherein theimage of the third type is a phase shift image.
 33. The method of claim23, further comprising: generating, based on the training image of thefirst type, an image of a fourth type.
 34. The method of claim 33,wherein the image of the fourth type comprises segmentation data. 35.The method of claim 23, wherein the training image of the first type iscaptured using a microscope according to a first illumination scheme.36. The method of claim 35, wherein the first illumination schemecomprises one or more illumination patterns.
 37. The method of claim 35,wherein the training image of the first type is part of a bright-fieldimage array.
 38. The method of claim 35, wherein the plurality oftraining images is a first plurality of training images, the methodfurther comprising: based on the comparison, identifying a secondillumination scheme; obtaining a second plurality of training imagescomprising one or more images of the first type, wherein the one or moreimages of the first type are obtained based on the second illuminationscheme; training the machine-learning model based on the secondplurality of training images.
 39. The method of claim 35, furthercomprising: obtaining, using a microscope, a plurality of images of thefirst type; and generating, based on the obtained plurality of images, aplurality of synthetic images of the second type using themachine-learning model.
 40. The method of claim 39, further comprising:training a classifier based on the plurality of synthetic images of thesecond type.
 41. The method of claim 40, wherein the microscope is afirst microscope, wherein the classifier is a first classifier, furthercomprising: obtaining, using a second microscope, a plurality of imagesof the second type; training a second classifier based on the pluralityof images of the second type; comparing performance of the firstclassifier and the second classifier.
 42. The method of claim 41,wherein the second microscope is a fluorescence microscope. 43-84.(canceled)
 85. A method of processing images of a biological sample toobtain one or more output images, comprising: obtaining a plurality ofimages of the biological sample using a plurality of configurations of aSLM of an optical system, wherein the SLM is located in an optical pathbetween the biological sample and an image recording device; andinputting the plurality of images of the biological sample into atrained machine-learning model to obtain the one or more outputs imagesor one or more classification outputs. 86-141. (canceled)
 142. Themethod of claim 22, wherein the image of the biological sample isilluminated using an illumination pattern of an illumination source,wherein the illumination pattern is determined by: training aclassification model configured to receive an input image and output aclassification result, training, using the trained classification model,the machine-learning model having an plurality of weights correspondingto a plurality of illumination settings, and identifying theillumination pattern based on the plurality of weights of the trainedmachine-learning model. 143-144. (canceled)
 145. The method of claim142, wherein the illumination source comprises an array of illuminationemitters.
 146. The method of claim 145, wherein the illumination sourceis a LED array.
 147. The method of claim 142, wherein the illuminationpattern indicates whether each illumination emitter is turned on or offand the intensity of each illumination emitter.
 148. The method of claim145, wherein each illumination setting of the plurality of illuminationsettings corresponds to a respective illumination emitter of theillumination source; and wherein each weight corresponds to an intensityof the respective illumination emitter.
 149. The method of claim 142,wherein the classification model is configured to receive an input phaseimage or an input fluorescence image and output a classification resultindicative of one class out of a plurality of pre-defined classes.150-161. (canceled)
 162. A method of evaluating a treatment with respectto a disease of interest, comprising: receiving a first plurality ofimages depicting a first set of healthy biological samples not affectedby the disease of interest; receiving a second plurality of imagesdepicting a second set of untreated biological samples affected by thedisease of interest; receiving a third plurality of images depicting athird set of treated biological samples affected by the disease ofinterest and treated by the treatment; inputting the first plurality ofimages into a trained machine-learning model to obtain a first pluralityof enhanced images; inputting the second plurality of images into thetrained machine-learning model to obtain a second plurality of enhancedimages; inputting the third plurality of images into the trainedmachine-learning model to obtain a third plurality of enhanced images;comparing the first plurality of enhanced images, the second pluralityof enhanced images, and the third plurality of enhanced images toevaluate the treatment. 163-182. (canceled)
 183. A method for training amachine-learning model to generate images of biological samples,comprising: obtaining a plurality of training images comprising: atraining image of a first type, and a training image of a second type;generating, based on the training image of the first type, a pluralityof wavelet coefficients using the machine-learning model; generating,based on the plurality of wavelet coefficients, a synthetic image of thesecond type; comparing the synthetic image of the second type with thetraining image of the second type; and updating the machine-learningmodel based on the comparison.